diff --git a/docs/RegisterInstructionOptimizationReport.md b/docs/RegisterInstructionOptimizationReport.md new file mode 100644 index 0000000..f744f86 --- /dev/null +++ b/docs/RegisterInstructionOptimizationReport.md @@ -0,0 +1,171 @@ +# Register Instruction Optimization Report + +This report tracks narrowly scoped generated-code opportunities in SimdLib's +register implementation. An item is complete only when its semantics are covered +by tests and representative optimized output demonstrates the intended instruction +property. Instruction-count changes are local evidence and do not establish an +application-level speedup. + +## Candidate summary + +| Item | Area | Intended improvement | Status | +|---:|---|---|---| +| 1 | Runtime integer lane insertion | Replace 256-bit half selection with one native comparison and blend using static lane indices. | Complete. | +| 2 | Runtime complete-register byte shifts | Eliminate fixed byte-shuffle selector preparation in unoptimized output. | Complete; existing implementation is optimal. | +| 3 | Signed minimum-position reductions | Replace procedural lane-index construction with native constant expressions. | Complete. | +| 4 | `RegisterMask::all()` | Use complete-register containment for 16-bit predicate lanes. | Complete. | +| 5 | Runtime 128-bit whole-register bit shifts | Execute only the runtime count range selected by the caller. | Complete; distribution-sensitive tradeoff documented. | + +## 1. Runtime integer lane insertion + +The 128-bit signed and unsigned integer specializations create fixed lane-index +vectors with `_mm_setr_epi*` or `_mm_set_epi64x`. Optimized MSVC output already +uses those values directly from the constant pool, so changing their source +expression does not improve the generated instructions. + +The original 256-bit integer specializations instead branch on the selected half, +extract or cast that half, perform a 128-bit comparison and blend, and then +recombine the result. Their native AVX2 implementations now use static native +lane-index constants, compare the runtime index across the complete 256-bit +register, and perform one byte blend. This removes the control flow and +half-register reconstruction while preserving the existing 128-bit path. + +Direct use of `make_static_register()` at these eight sites produced correct +constant-pool operands but also caused MSVC to emit an otherwise unused aligned +stack frame in the real API instantiation. The native `_mm256_setr_epi*` +expressions produce the same constant-pool operands without that frame and are +therefore retained here. + +Required evidence: + +- existing runtime insertion tests pass for every supported integer element type + and both register widths; +- representative optimized 128-bit and 256-bit callers contain no stack-built + lane-index vectors; +- the fixed lane indices appear as constant-pool operands or an equally compact + compiler-generated form; +- 256-bit callers contain no half-selection branch, extraction, or reinsertion. + +Validation results: + +- the MSVC Release SSE4.2 runtime-insertion test passed 66 assertions; +- the MSVC Release AVX2 runtime-insertion test passed 132 assertions; +- the SSE4.2 and AVX2 strict generated-code gates both passed; +- all eight signed and unsigned AVX2 integer forms use a constant-pool + `VPCMPEQ*`, one full-width `VPBLENDVB`, and no stack frame, branch, + half-register extraction, reinsertion, helper call, or spill; +- representative `uint32x8` output changed from a branched half-register + sequence of approximately `0x5a` bytes to a straight-line `0x22`-byte + sequence. + +## 2. Runtime complete-register byte shifts + +The 128-bit runtime byte-shift extensions use fixed ascending or biased byte +indices to construct `PSHUFB` controls. Optimized MSVC output loads both +`_mm_setr_epi8` values directly from the constant pool, but SimdLib evaluates +instruction implementations against unoptimized output so correctness does not +depend on optimizer recognition. The `/Od` paths therefore remain authoritative. + +The left-shift control computes `indices - counts`; its constant must be the +destination of the two-operand SSE subtraction and therefore requires a register +load. The right-shift control computes `biasedIndices + counts`, so reversing the +commutative intrinsic operands was evaluated as a way to fold the constant into +`PADDB`. MSVC canonicalizes both source expressions to the same constant-first +sequence, leaving the separate constant-pool load unchanged. + +The `/Od` audit confirms that the existing local `_mm_setr_epi8` expressions each +produce one aligned constant-pool load, with no byte-wise selector stores and no +selector-construction helper call. An explicit aligned static array produces the +same instruction, while `make_static_register()` introduces calls and procedural +stack construction when optimization is disabled. A static native `__m128i` +requires dynamic initialization and is also unsuitable. + +The remaining `/Od` stack traffic belongs to ordinary intrinsic temporaries and +the public API or Register call boundaries rather than selector preparation. +Item 2 therefore requires no source change: the existing expression already +produces the optimal selector instruction under the unoptimized policy. + +## 3. Signed minimum-position reductions + +Signed 8-, 16-, 32-, and 64-bit minimum-position implementations previously +constructed their 128-bit lane-index vectors through `register_from_values()`. +Under MSVC `/Od`, that path emitted 16 byte, 8 word, 4 double-word, or 2 +quad-word immediate stores respectively before loading the completed vector. + +The implementations now express those constants with the corresponding native +`_mm_setr_epi*` intrinsic, or `_mm_set_epi64x` for 64-bit lanes. MSVC lowers +each expression to one constant-pool vector load even without optimization and +does not call a construction helper. The 256-bit reductions divide their +inputs into 128-bit halves and therefore inherit the improved construction for +each half without requiring a separate 256-bit constant. + +Validation results: + +- the focused SSE4.2 position test passed 32 assertions; +- the focused AVX2 position test passed 64 assertions across 128- and 256-bit + registers; +- fresh MSVC `/Od /Ob0` SSE4.2 and AVX2 output uses one `MOVDQA` or `VMOVDQU` + constant-pool load for each signed lane-index vector; +- no scalar immediate lane stores or register-construction helper calls remain; +- remaining unoptimized spills and reloads are ordinary debug lowering after + the constant has been loaded. + +## 4. `RegisterMask::all()` + +Canonical predicate lanes permit a complete-register containment test against an +all-one vector. The 16-bit specializations now use `PTEST` or `VPTEST` for this +purpose, avoiding the byte shuffle, byte movemask, and scalar mask comparison +required by their previous compact-mask path. Constant evaluation and non-x86 +backends retain the portable compact-mask comparison, and other x86 element +widths retain their existing paths because their direct movemask instructions +are already competitive. + +The runtime path creates the all-one predicate by comparing the byte view with +itself. This is expressed directly with the width-appropriate native intrinsics +so MSVC `/Od /Ob0` does not introduce calls to the constant-evaluation, +bit-cast, comparison, test, or compact-mask helpers. Optimized MSVC folds the +all-one value into the memory operand of `PTEST` or `VPTEST`. + +Validation results: + +- the focused SSE4.2 mask test passed 558 assertions; +- the focused AVX2 mask test passed 1,342 assertions; +- fresh MSVC `/Od /Ob0` output uses `PCMPEQB` plus `PTEST` for 128-bit masks and + `VPCMPEQB` plus `VPTEST` for 256-bit masks, with no reduction-helper calls; +- optimized 128-bit output decreased from 25 bytes to 15 bytes and removes + `PSHUFB`, `PMOVMSKB`, and the scalar comparison; +- optimized 256-bit output decreased from 44 bytes to 18 bytes and removes + `VPSHUFB`, `VPMOVMSKB`, scalar byte-mask compaction, and the scalar comparison; +- signed and unsigned 16-bit masks produce identical reduction sequences, while + 8-, 32-, and 64-bit integer and floating-point paths remain unchanged. + +## 5. Runtime 128-bit whole-register bit shifts + +The previous runtime whole-register shifts clamped the count, computed results +for counts below and above 64 bits, and combined them. The implementations now +branch according to the public count contract and compute only the selected +identity, zero, below-64, exactly-64, or above-64 result. This mirrors the static +shift structure and avoids calculating a range result that will be discarded. + +The change deliberately favors predictable or range-skewed counts. A matched +MSVC `/O2 /arch:AVX2` microbenchmark used chained results, 65,536-count inputs, +256 repetitions per sample, 11 alternating baseline/candidate rounds, and three +process runs on an Intel Family 6 Model 151 processor. The candidate took +approximately 51-73% of the previous time for mostly-below-64, +mostly-above-64, and predictable mixed distributions. Uniformly randomized +counts from 1 through 127 instead took approximately 2.27-2.51 times as long +because the range branch was unpredictable. This is a distribution-sensitive +local tradeoff, not evidence of an application-level speedup. + +Validation results: + +- five focused SSE4.2 and AVX2 runtime suites passed, including exhaustive + runtime counts from 0 through 127 and negative and above-width boundaries; +- the API and Register constant-evaluation probes compiled successfully; +- MSVC `/Od /Ob0` helper size decreased from `0x1CE` to `0x173` bytes for SSE4.2 + and from `0x1D2` to `0x176` bytes for AVX2; +- optimized helper size decreased from `0x58` to `0x50` bytes for SSE4.2 and to + `0x53` bytes for AVX2, although the static instruction count increased from 23 + to 25 or 26 because all range branches are present; +- the selected optimized runtime paths execute approximately 4-16 instructions + rather than the complete branchless calculation. diff --git a/include/SimdLib/Detail/Extensions.h b/include/SimdLib/Detail/Extensions.h index 676b07d..4aa67b4 100644 --- a/include/SimdLib/Detail/Extensions.h +++ b/include/SimdLib/Detail/Extensions.h @@ -14,6 +14,7 @@ #endif #include #include +#include namespace SimdLib::Detail { @@ -86,71 +87,6 @@ constexpr Element SIMD_FLAGS(Neither, ForceInline) register_get_constexpr(const #endif } -/** - * @brief Reads one runtime lane through the portable native-register representation. - * @tparam Element Scalar lane type. - * @tparam Vector Compiler-native register type. - * @param value Source register. - * @param index Selected lane index. - * @return Selected scalar lane. - * @note This fallback may materialize addressable storage and must not be used by register-only operation paths. - */ -template - requires std::is_arithmetic_v && (sizeof(Vector) % sizeof(Element) == 0) -Element SIMD_FLAGS(Neither, ForceInline) register_get(const Vector value, const std::size_t index) noexcept -{ -#if SIMDLIB_COMPILER_MSVC - if constexpr (sizeof(Vector) == 16) - { - if constexpr (std::is_integral_v && sizeof(Element) == 1 && std::is_unsigned_v) - return value.m128i_u8[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 1) - return value.m128i_i8[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 2 && std::is_unsigned_v) - return value.m128i_u16[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 2) - return value.m128i_i16[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 4 && std::is_unsigned_v) - return value.m128i_u32[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 4) - return value.m128i_i32[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 8 && std::is_unsigned_v) - return value.m128i_u64[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 8) - return value.m128i_i64[index]; - else if constexpr (std::same_as) - return value.m128_f32[index]; - else - return value.m128d_f64[index]; - } - else - { - if constexpr (std::is_integral_v && sizeof(Element) == 1 && std::is_unsigned_v) - return value.m256i_u8[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 1) - return value.m256i_i8[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 2 && std::is_unsigned_v) - return value.m256i_u16[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 2) - return value.m256i_i16[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 4 && std::is_unsigned_v) - return value.m256i_u32[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 4) - return value.m256i_i32[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 8 && std::is_unsigned_v) - return value.m256i_u64[index]; - else if constexpr (std::is_integral_v && sizeof(Element) == 8) - return value.m256i_i64[index]; - else if constexpr (std::same_as) - return value.m256_f32[index]; - else - return value.m256d_f64[index]; - } -#else - return std::bit_cast>(value)[index]; -#endif -} - /** * @brief Replaces one lane through the portable constant-evaluation representation. * @tparam Element Scalar lane type. @@ -219,23 +155,111 @@ constexpr void SIMD_FLAGS(Neither, ForceInline) register_set_constexpr(Vector &v #endif } +/** + * @brief Constructs a native register from a complete forward-order lane array. + * @tparam Vector Compiler-native register type. + * @tparam Element Scalar lane type. + * @tparam Count Complete logical lane count. + * @param lanes Source lanes in increasing register-lane order. + * @return Native register containing every source lane. + */ template requires(sizeof(Vector) == sizeof(Element) * Count) -constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_from_array(const std::array &lanes) noexcept +constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_from_array_constexpr(const std::array &lanes) noexcept { +#if SIMDLIB_COMPILER_MSVC Vector result{}; for (std::size_t index = 0; index < Count; ++index) - { register_set_constexpr(result, index, lanes[index]); - } return result; +#else + return std::bit_cast(lanes); +#endif } template requires(sizeof(Vector) == sizeof(Element) * sizeof...(Args)) && (std::convertible_to && ...) constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_from_values(Args &&...values) noexcept { - return register_from_array(std::array{static_cast(values)...}); + return register_from_array_constexpr(std::array{static_cast(values)...}); +} + +/** @brief Scalar element type generated by one static-register expression. */ +template using static_register_element_t = std::remove_cvref_t())>; + +/** + * @brief Aligned, typed lane storage shared by constant-evaluated and runtime static-register construction. + * @tparam Implementation Native backend exposing its resulting `register_t`. + * @tparam expression Stateless compile-time lane generator. + * @tparam indices Complete forward-order lane index sequence. + */ +template +alignas(sizeof(typename Implementation::register_t)) inline constexpr static_register_element_t + static_register_lanes[sizeof...(indices)]{expression.template operator()()...}; + +/** + * @brief Expands a static lane generator over the complete inferred register geometry. + * @tparam Implementation Native backend exposing `register_t` and the runtime `setr` constructor. + * @tparam expression Stateless compile-time lane generator invoked as `expression.operator()()`. + * @tparam indices Internally generated forward-order lane indices. + * @return Native register containing the generated lane sequence. + */ +template +constexpr auto SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) make_static_register_from_indices(std::index_sequence) noexcept +{ + using element_type = std::remove_cvref_t())>; + using vector_type = typename Implementation::register_t; + constexpr auto &lanes = static_register_lanes; + if (std::is_constant_evaluated()) + { + vector_type result{}; + (register_set_constexpr(result, indices, lanes[indices]), ...); + return result; + } + +#if SIMDLIB_TARGET_X86 + // Load the typed static data directly so runtime construction never evaluates or stores individual lanes. + if constexpr (sizeof(vector_type) == 16) + { + if constexpr (std::is_integral_v) + return _mm_load_si128(reinterpret_cast(lanes)); + else if constexpr (std::same_as) + return _mm_load_ps(lanes); + else + return _mm_load_pd(lanes); + } + else + { + if constexpr (std::is_integral_v) + return _mm256_load_si256(reinterpret_cast(lanes)); + else if constexpr (std::same_as) + return _mm256_load_ps(lanes); + else + return _mm256_load_pd(lanes); + } +#else + return Implementation::setr(expression.template operator()()...); +#endif +} + +/** + * @brief Constructs a native register from one stateless compile-time lane generator. + * @tparam Implementation Native backend exposing its native `register_t`. + * @tparam Expression Stateless generator type with a templated zero-argument call operator. + * @param generator Generator used only for type deduction; its stateless type defines the compile-time expression. + * @return Native register containing one generated value per inferred logical lane. + * @remarks The helper owns register-width discovery and index-sequence expansion. Constant evaluation uses the portable lane writer; runtime loads the same typed, aligned static lane data. + */ +template + requires std::is_empty_v> && std::is_default_constructible_v> +constexpr auto SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) make_static_register([[maybe_unused]] Expression generator) noexcept +{ + using expression_type = std::remove_cvref_t; + constexpr expression_type expression{}; + using element_type = std::remove_cvref_t())>; + using vector_type = typename Implementation::register_t; + constexpr std::size_t lane_count = sizeof(vector_type) / sizeof(element_type); + return make_static_register_from_indices(std::make_index_sequence{}); } /** @brief Constructs a constant-evaluated native register with every lane set to one value. @@ -250,7 +274,7 @@ constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_from_repeated_value(c { std::array lanes{}; lanes.fill(value); - return register_from_array(lanes); + return register_from_array_constexpr(lanes); } template constexpr auto SIMD_FLAGS(Neither, ForceInline) register_to_array(const Vector value) noexcept @@ -292,16 +316,16 @@ constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_insert_constexpr(Vect return value; } -/** @brief Emulates an immediate-controlled lane blend with a runtime scalar mask. +/** @brief Evaluates an immediate-controlled lane blend through the portable constant-evaluation representation. * @tparam Element Logical lane type. * @tparam Vector Native register type. * @param lhs Source for lanes whose control bits are clear. * @param rhs Source for lanes whose control bits are set. - * @param mask Runtime control byte. + * @param mask Immediate control represented during constant evaluation. * @return Register containing the selected lanes. */ template -constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_blend_slow(Vector lhs, const Vector rhs, const unsigned int mask) noexcept +constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_blend_constexpr(Vector lhs, const Vector rhs, const unsigned int mask) noexcept { constexpr std::size_t count = sizeof(Vector) / sizeof(Element); for (std::size_t index = 0; index < count; ++index) @@ -351,7 +375,10 @@ constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_shuffle_float_slow(co result[lane + 2] = right[lane + ((control >> 4) & 0x3u)]; result[lane + 3] = right[lane + ((control >> 6) & 0x3u)]; } - return register_from_array(result); + if constexpr (sizeof(Vector) == 16) + return _mm_loadu_ps(result.data()); + else + return _mm256_loadu_ps(result.data()); } /** @brief Emulates a double shuffle with a runtime control byte. @@ -373,7 +400,10 @@ constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_shuffle_double_slow(c result[lane] = left[lane + (lane_control & 0x1u)]; result[lane + 1] = right[lane + ((lane_control >> 1) & 0x1u)]; } - return register_from_array(result); + if constexpr (sizeof(Vector) == 16) + return _mm_loadu_pd(result.data()); + else + return _mm256_loadu_pd(result.data()); } /** @brief Emulates a 32-bit shuffle with a runtime control byte. @@ -394,7 +424,7 @@ constexpr Vector SIMD_FLAGS(InOut, RegisterOnly, ForceInline) register_shuffle_3 for (std::size_t index = 0; index < 4; ++index) result[lane + index] = source[lane + ((control >> (index * 2)) & 0x3u)]; } - return register_from_array(result); + return register_from_array_constexpr(result); } const int index0 = static_cast(control & 0x3u); @@ -436,7 +466,7 @@ constexpr Vector SIMD_FLAGS(InOut, RegisterOnly, ForceInline) for (std::size_t index = 0; index < 4; ++index) result[base + index] = source[base + ((control >> (index * 2)) & 0x3u)]; } - return register_from_array(result); + return register_from_array_constexpr(result); } const int index0 = static_cast(control & 0x3u); @@ -466,16 +496,6 @@ constexpr Vector SIMD_FLAGS(InOut, RegisterOnly, ForceInline) return _mm256_shuffle_epi8(value, _mm256_broadcastsi128_si256(byte_indices)); } -template -constexpr Vector SIMD_FLAGS(Neither, ForceInline) register_transform_binary(const Vector lhs, const Vector rhs, Operation &&operation) noexcept -{ - constexpr std::size_t count = sizeof(Vector) / sizeof(Element); - std::array result{}; - for (std::size_t index = 0; index < count; ++index) - result[index] = static_cast(operation(register_get_constexpr(lhs, index), register_get_constexpr(rhs, index))); - return register_from_array(result); -} - #if SIMDLIB_HAS_SSE42 #pragma region 128bit Complete-Register Byte Shift Extensions @@ -1535,16 +1555,23 @@ __m128i SIMD_FLAGS(InOut, ForceInline) _ext_max_epu64(__m128i lhs, __m128i rhs) * @param lhs Source register interpreted as one unsigned 128-bit bit string. * @param shift Runtime count; nonpositive counts are identity and counts of at least 128 produce zero. * @return Shifted register with zero-filled low bits. + * @remarks Count ranges are handled independently so the runtime path computes only the selected cross-lane transfer. */ __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) _ext128_shift_bits_left_slow(const __m128i lhs, const int shift) noexcept { - const __m128i count = _mm_min_epi32(_mm_max_epi32(_mm_cvtsi32_si128(shift), _mm_setzero_si128()), _mm_cvtsi32_si128(128)); - const __m128i midpoint = _mm_cvtsi32_si128(64); - const __m128i complement = _mm_sub_epi64(midpoint, count); - const __m128i excess = _mm_sub_epi64(count, midpoint); - const __m128i low_range = _mm_or_si128(_mm_sll_epi64(lhs, count), _mm_slli_si128(_mm_srl_epi64(lhs, complement), 8)); - const __m128i high_range = _mm_sll_epi64(_mm_slli_si128(lhs, 8), excess); - return _mm_or_si128(low_range, high_range); + if (shift <= 0) + return lhs; + if (shift >= 128) + return _mm_setzero_si128(); + if (shift < 64) + { + const __m128i count = _mm_cvtsi32_si128(shift); + const __m128i complement = _mm_cvtsi32_si128(64 - shift); + return _mm_or_si128(_mm_sll_epi64(lhs, count), _mm_slli_si128(_mm_srl_epi64(lhs, complement), 8)); + } + if (shift == 64) + return _mm_slli_si128(lhs, 8); + return _mm_sll_epi64(_mm_slli_si128(lhs, 8), _mm_cvtsi32_si128(shift - 64)); } /** @@ -1573,16 +1600,23 @@ template __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) _ext12 * @param lhs Source register interpreted as one unsigned 128-bit bit string. * @param shift Runtime count; nonpositive counts are identity and counts of at least 128 produce zero. * @return Shifted register with zero-filled high bits. + * @remarks Count ranges are handled independently so the runtime path computes only the selected cross-lane transfer. */ __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) _ext128_shift_bits_right_slow(const __m128i lhs, const int shift) noexcept { - const __m128i count = _mm_min_epi32(_mm_max_epi32(_mm_cvtsi32_si128(shift), _mm_setzero_si128()), _mm_cvtsi32_si128(128)); - const __m128i midpoint = _mm_cvtsi32_si128(64); - const __m128i complement = _mm_sub_epi64(midpoint, count); - const __m128i excess = _mm_sub_epi64(count, midpoint); - const __m128i low_range = _mm_or_si128(_mm_srl_epi64(lhs, count), _mm_srli_si128(_mm_sll_epi64(lhs, complement), 8)); - const __m128i high_range = _mm_srl_epi64(_mm_srli_si128(lhs, 8), excess); - return _mm_or_si128(low_range, high_range); + if (shift <= 0) + return lhs; + if (shift >= 128) + return _mm_setzero_si128(); + if (shift < 64) + { + const __m128i count = _mm_cvtsi32_si128(shift); + const __m128i complement = _mm_cvtsi32_si128(64 - shift); + return _mm_or_si128(_mm_srl_epi64(lhs, count), _mm_srli_si128(_mm_sll_epi64(lhs, complement), 8)); + } + if (shift == 64) + return _mm_srli_si128(lhs, 8); + return _mm_srl_epi64(_mm_srli_si128(lhs, 8), _mm_cvtsi32_si128(shift - 64)); } /** diff --git a/include/SimdLib/Detail/Implementations.h b/include/SimdLib/Detail/Implementations.h index a675e61..8172ab3 100644 --- a/include/SimdLib/Detail/Implementations.h +++ b/include/SimdLib/Detail/Implementations.h @@ -212,8 +212,10 @@ template [[nodiscard]] consteval int en template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects bytes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -226,69 +228,69 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 16 && ((indices < 16) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi8(lhs, _mm_setr_epi8(static_cast(indices)...)); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi8(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m128i lhsWideLo = _mm_cvtepi8_epi16(lhs); - const __m128i rhsWideLo = _mm_cvtepi8_epi16(rhs); - const __m128i lhsWideHi = _mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8)); - const __m128i rhsWideHi = _mm_cvtepi8_epi16(_mm_srli_si128(rhs, 8)); + const register_t lhsWideLo = _mm_cvtepi8_epi16(lhs); + const register_t rhsWideLo = _mm_cvtepi8_epi16(rhs); + const register_t lhsWideHi = _mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8)); + const register_t rhsWideHi = _mm_cvtepi8_epi16(_mm_srli_si128(rhs, 8)); return _mm_hadd_epi16(_mm_mullo_epi16(lhsWideLo, rhsWideLo), _mm_mullo_epi16(lhsWideHi, rhsWideHi)); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext_mul_epi8(lhs, rhs); } /** @brief Divides corresponding signed 8-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epi8(lhs, rhs); } /** @brief Computes corresponding signed 8-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epi8(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { - auto sqrt16 = [](__m128i values) noexcept + auto sqrt16 = [](register_t values) noexcept { - const __m128i lo32 = _mm_cvtepi16_epi32(values); - const __m128i hi32 = _mm_cvtepi16_epi32(_mm_srli_si128(values, 8)); - const __m128i loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(lo32))); - const __m128i hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(hi32))); + const register_t lo32 = _mm_cvtepi16_epi32(values); + const register_t hi32 = _mm_cvtepi16_epi32(_mm_srli_si128(values, 8)); + const register_t loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(lo32))); + const register_t hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(hi32))); return _mm_packs_epi32(loRoots, hiRoots); }; - const __m128i lo16 = sqrt16(_mm_cvtepi8_epi16(lhs)); - const __m128i hi16 = sqrt16(_mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8))); + const register_t lo16 = sqrt16(_mm_cvtepi8_epi16(lhs)); + const register_t hi16 = sqrt16(_mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8))); return _mm_packs_epi16(lo16, hi16); } /** @brief Computes the unchecked group magnitude in lane zero; all other lanes are unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - const __m128i low = _mm_cvtepi8_epi16(lhs); - const __m128i high = _mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8)); - __m128i total = _mm_add_epi16(_mm_mullo_epi16(low, low), _mm_mullo_epi16(high, high)); + const register_t low = _mm_cvtepi8_epi16(lhs); + const register_t high = _mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8)); + register_t total = _mm_add_epi16(_mm_mullo_epi16(low, low), _mm_mullo_epi16(high, high)); total = _mm_hadd_epi16(total, total); total = _mm_hadd_epi16(total, total); total = _mm_hadd_epi16(total, total); @@ -297,13 +299,13 @@ template <> struct SimdImpl128 } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t threshold = maximum * maximum + maximum + 1; - const __m128i low = _mm_cvtepi8_epi16(lhs); - const __m128i high = _mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8)); - __m128i pairSums = _mm_add_epi32(_mm_madd_epi16(low, low), _mm_madd_epi16(high, high)); + const register_t low = _mm_cvtepi8_epi16(lhs); + const register_t high = _mm_cvtepi8_epi16(_mm_srli_si128(lhs, 8)); + register_t pairSums = _mm_add_epi32(_mm_madd_epi16(low, low), _mm_madd_epi16(high, high)); pairSums = _mm_hadd_epi32(pairSums, pairSums); pairSums = _mm_hadd_epi32(pairSums, pairSums); const std::uint64_t total = static_cast(_mm_cvtsi128_si32(pairSums)); @@ -313,16 +315,16 @@ template <> struct SimdImpl128 } /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { - constexpr __m128i indices = register_from_values<__m128i, std::int8_t>(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); - __m128i values = lhs; - __m128i positions = indices; + const register_t indices = _mm_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); + register_t values = lhs; + register_t positions = indices; auto reduce = [&]() noexcept { - const __m128i shiftedValues = _mm_bsrli_si128(values, offset); - const __m128i shiftedIndices = _mm_bsrli_si128(positions, offset); - const __m128i less = _mm_cmpgt_epi8(values, shiftedValues); + const register_t shiftedValues = _mm_bsrli_si128(values, offset); + const register_t shiftedIndices = _mm_bsrli_si128(positions, offset); + const register_t less = _mm_cmpgt_epi8(values, shiftedValues); values = _mm_blendv_epi8(values, shiftedValues, less); positions = _mm_blendv_epi8(positions, shiftedIndices, less); }; @@ -333,89 +335,89 @@ template <> struct SimdImpl128 return _mm_insert_epi8(values, _mm_extract_epi8(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm_abs_epi8(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi8(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_epi8(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_epi8(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _ext_slli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _ext_srli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext_srai_epx8(lhs, rhs); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm_adds_epi8(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm_subs_epi8(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi8(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm_set_epi8(static_cast(args)...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm_setr_epi8(static_cast(args)...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm_cmpgt_epi8(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtepi8_epi16(lhs, rhs); } @@ -463,7 +465,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 16)`. * @return Selected scalar lane. */ - static int8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static int8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Signed 8-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -505,12 +507,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected signed 8-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int8_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int8_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 8-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int8_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int8_t rhs) noexcept { return _mm_insert_epi8(lhs, static_cast(rhs), index); } @@ -521,33 +523,33 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 16)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const int8_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int8_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Signed 8-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); - const __m128i selected_lane = _mm_cmpeq_epi8(lane_indices, _mm_set1_epi8(static_cast(index))); + const register_t lane_indices = _mm_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); + const register_t selected_lane = _mm_cmpeq_epi8(lane_indices, _mm_set1_epi8(static_cast(index))); return _mm_blendv_epi8(lhs, _mm_set1_epi8(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi8(lhs, rhs); } // misc /** @brief Shuffles bytes through the native runtime selector-register instruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept - requires(std::same_as && std::same_as) + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept + requires(std::same_as && std::same_as) { return _mm_shuffle_epi8(lhs, rhs); } /** @brief Selects bytes through the native runtime mask-register operation. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const __m128i lhs, const __m128i rhs, const __m128i mask) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const register_t lhs, const register_t rhs, const register_t mask) noexcept { return register_blend_bytes(lhs, rhs, mask); } @@ -559,8 +561,10 @@ template <> struct SimdImpl128 template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects bytes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -573,69 +577,69 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 16 && ((indices < 16) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi8(lhs, _mm_setr_epi8(static_cast(indices)...)); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi8(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m128i lhsWideLo = _mm_cvtepu8_epi16(lhs); - const __m128i rhsWideLo = _mm_cvtepu8_epi16(rhs); - const __m128i lhsWideHi = _mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8)); - const __m128i rhsWideHi = _mm_cvtepu8_epi16(_mm_srli_si128(rhs, 8)); + const register_t lhsWideLo = _mm_cvtepu8_epi16(lhs); + const register_t rhsWideLo = _mm_cvtepu8_epi16(rhs); + const register_t lhsWideHi = _mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8)); + const register_t rhsWideHi = _mm_cvtepu8_epi16(_mm_srli_si128(rhs, 8)); return _mm_hadd_epi16(_mm_mullo_epi16(lhsWideLo, rhsWideLo), _mm_mullo_epi16(lhsWideHi, rhsWideHi)); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext_mul_epi8(lhs, rhs); } /** @brief Divides corresponding unsigned 8-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epu8(lhs, rhs); } /** @brief Computes corresponding unsigned 8-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epu8(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { - auto sqrt16 = [](__m128i values) noexcept + auto sqrt16 = [](register_t values) noexcept { - const __m128i lo32 = _mm_cvtepu16_epi32(values); - const __m128i hi32 = _mm_cvtepu16_epi32(_mm_srli_si128(values, 8)); - const __m128i loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(lo32))); - const __m128i hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(hi32))); + const register_t lo32 = _mm_cvtepu16_epi32(values); + const register_t hi32 = _mm_cvtepu16_epi32(_mm_srli_si128(values, 8)); + const register_t loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(lo32))); + const register_t hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(hi32))); return _mm_packus_epi32(loRoots, hiRoots); }; - const __m128i lo16 = sqrt16(_mm_cvtepu8_epi16(lhs)); - const __m128i hi16 = sqrt16(_mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8))); + const register_t lo16 = sqrt16(_mm_cvtepu8_epi16(lhs)); + const register_t hi16 = sqrt16(_mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8))); return _mm_packus_epi16(lo16, hi16); } /** @brief Computes the unchecked group magnitude in lane zero; all other lanes are unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - const __m128i low = _mm_cvtepu8_epi16(lhs); - const __m128i high = _mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8)); - __m128i total = _mm_add_epi16(_mm_mullo_epi16(low, low), _mm_mullo_epi16(high, high)); + const register_t low = _mm_cvtepu8_epi16(lhs); + const register_t high = _mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8)); + register_t total = _mm_add_epi16(_mm_mullo_epi16(low, low), _mm_mullo_epi16(high, high)); total = _mm_hadd_epi16(total, total); total = _mm_hadd_epi16(total, total); total = _mm_hadd_epi16(total, total); @@ -644,13 +648,13 @@ template <> struct SimdImpl128 } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t threshold = maximum * maximum + maximum + 1; - const __m128i low = _mm_cvtepu8_epi16(lhs); - const __m128i high = _mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8)); - __m128i pairSums = _mm_add_epi32(_mm_madd_epi16(low, low), _mm_madd_epi16(high, high)); + const register_t low = _mm_cvtepu8_epi16(lhs); + const register_t high = _mm_cvtepu8_epi16(_mm_srli_si128(lhs, 8)); + register_t pairSums = _mm_add_epi32(_mm_madd_epi16(low, low), _mm_madd_epi16(high, high)); pairSums = _mm_hadd_epi32(pairSums, pairSums); pairSums = _mm_hadd_epi32(pairSums, pairSums); const std::uint64_t total = static_cast(_mm_cvtsi128_si32(pairSums)); @@ -659,18 +663,27 @@ template <> struct SimdImpl128 return magnitude_checked_result(result, overflow); } + /** @brief Returns the unsigned-byte lane indices as one compile-time-generated native register. */ + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) lane_indices() noexcept + { + return make_static_register>([]() constexpr noexcept + { + return static_cast(index); + }); + } + /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { - constexpr __m128i indices = register_from_values<__m128i, std::uint8_t>(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); - const __m128i signBit = _mm_set1_epi8(static_cast(0x80)); - __m128i values = lhs; - __m128i positions = indices; + const register_t indices = lane_indices(); + const register_t signBit = _mm_set1_epi8(static_cast(0x80)); + register_t values = lhs; + register_t positions = indices; auto reduce = [&]() noexcept { - const __m128i shiftedValues = _mm_bsrli_si128(values, offset); - const __m128i shiftedIndices = _mm_bsrli_si128(positions, offset); - const __m128i less = _mm_cmpgt_epi8(_mm_xor_si128(values, signBit), _mm_xor_si128(shiftedValues, signBit)); + const register_t shiftedValues = _mm_bsrli_si128(values, offset); + const register_t shiftedIndices = _mm_bsrli_si128(positions, offset); + const register_t less = _mm_cmpgt_epi8(_mm_xor_si128(values, signBit), _mm_xor_si128(shiftedValues, signBit)); values = _mm_blendv_epi8(values, shiftedValues, less); positions = _mm_blendv_epi8(positions, shiftedIndices, less); }; @@ -681,97 +694,99 @@ template <> struct SimdImpl128 return _mm_insert_epi8(values, _mm_extract_epi8(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm_abs_epi8(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi8(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_epu8(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_epu8(lhs, rhs); } /** @brief Computes lane-wise averages for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept { return _mm_avg_epu8(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _ext_slli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _ext_srli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext_srai_epx8(lhs, rhs); } // arithmetic (horizontal) - // static auto SIMD_FLAGS(InOut, ForceInline) hadd (auto lhs, auto rhs) noexcept { return _mm_hadd_epi8(lhs, rhs); } - // static auto SIMD_FLAGS(InOut, ForceInline) hsub (auto lhs, auto rhs) noexcept { return _mm_hsub_epi8(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) hadd (auto lhs, auto rhs) noexcept { return _mm_hadd_epi8(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) hsub (auto lhs, auto rhs) noexcept { return _mm_hsub_epi8(lhs, rhs); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm_adds_epu8(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm_subs_epu8(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + /** @brief Broadcasts one value to all unsigned 8-bit lanes. */ + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _ext_set1_epu8(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm_set_epi8(static_cast(args)...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + /** @brief Constructs unsigned 8-bit lanes in low-to-high logical order. */ + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm_setr_epi8(static_cast(args)...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext_cmpgt_epu8(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtepu8_epi16(lhs, rhs); } @@ -819,7 +834,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 16)`. * @return Selected scalar lane. */ - static uint8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static uint8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Unsigned 8-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -861,12 +876,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected unsigned 8-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint8_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint8_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 8-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint8_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint8_t rhs) noexcept { return _mm_insert_epi8(lhs, static_cast(rhs), index); } @@ -877,33 +892,33 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 16)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const uint8_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint8_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Unsigned 8-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); - const __m128i selected_lane = _mm_cmpeq_epi8(lane_indices, _mm_set1_epi8(static_cast(index))); + const register_t lane_indices = _mm_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); + const register_t selected_lane = _mm_cmpeq_epi8(lane_indices, _mm_set1_epi8(static_cast(index))); return _mm_blendv_epi8(lhs, _mm_set1_epi8(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi8(lhs, rhs); } // misc /** @brief Shuffles bytes through the native runtime selector-register instruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept - requires(std::same_as && std::same_as) + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept + requires(std::same_as && std::same_as) { return _mm_shuffle_epi8(lhs, rhs); } /** @brief Selects bytes through the native runtime mask-register operation. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const __m128i lhs, const __m128i rhs, const __m128i mask) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const register_t lhs, const register_t rhs, const register_t mask) noexcept { return register_blend_bytes(lhs, rhs, mask); } @@ -915,8 +930,10 @@ template <> struct SimdImpl128 template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects 16-bit lanes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -929,74 +946,74 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 8 && ((indices < 8) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi8(lhs, make_logical_shuffle_16_control(std::make_index_sequence<16>{})); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi16(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { return _mm_madd_epi16(lhs, rhs); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm_mullo_epi16(lhs, rhs); } /** @brief Divides corresponding signed 16-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epi16(lhs, rhs); } /** @brief Computes corresponding signed 16-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epi16(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { - const __m128i lo32 = _mm_cvtepi16_epi32(lhs); - const __m128i hi32 = _mm_cvtepi16_epi32(_mm_srli_si128(lhs, 8)); - const __m128i loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(lo32))); - const __m128i hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(hi32))); + const register_t lo32 = _mm_cvtepi16_epi32(lhs); + const register_t hi32 = _mm_cvtepi16_epi32(_mm_srli_si128(lhs, 8)); + const register_t loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(lo32))); + const register_t hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_mm_cvtepi32_ps(hi32))); return _mm_packs_epi32(loRoots, hiRoots); } /** @brief Computes the unchecked group magnitude in lane zero; all other lanes are unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - __m128i total = _mm_madd_epi16(lhs, lhs); + register_t total = _mm_madd_epi16(lhs, lhs); total = _mm_hadd_epi32(total, total); total = _mm_hadd_epi32(total, total); return _mm_cvtpd_epi32(_mm_sqrt_sd(_mm_cvtepi32_pd(total), _mm_cvtepi32_pd(total))); } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t threshold = maximum * maximum + maximum + 1; - const __m128i minimum = _mm_set1_epi16(std::numeric_limits::min()); + const register_t minimum = _mm_set1_epi16(std::numeric_limits::min()); if (_mm_movemask_epi8(_mm_cmpeq_epi16(lhs, minimum)) != 0) return magnitude_checked_result(maximum, true); - const __m128i pairSquares = _mm_madd_epi16(lhs, lhs); - const __m128i lowPairs = _mm_cvtepu32_epi64(pairSquares); - const __m128i highPairs = _mm_cvtepu32_epi64(_mm_srli_si128(pairSquares, 8)); - const __m128i pairTotals = _mm_add_epi64(lowPairs, highPairs); - const __m128i totalVector = _mm_add_epi64(pairTotals, _mm_srli_si128(pairTotals, 8)); + const register_t pairSquares = _mm_madd_epi16(lhs, lhs); + const register_t lowPairs = _mm_cvtepu32_epi64(pairSquares); + const register_t highPairs = _mm_cvtepu32_epi64(_mm_srli_si128(pairSquares, 8)); + const register_t pairTotals = _mm_add_epi64(lowPairs, highPairs); + const register_t totalVector = _mm_add_epi64(pairTotals, _mm_srli_si128(pairTotals, 8)); const std::uint64_t total = static_cast(_mm_cvtsi128_si64(totalVector)); const bool overflow = total >= threshold; const std::uint64_t result = overflow ? maximum : magnitude_round_sqrt_u64(total, maximum); @@ -1004,18 +1021,18 @@ template <> struct SimdImpl128 } /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { - constexpr __m128i indices = register_from_values<__m128i, std::int16_t>(0, 1, 2, 3, 4, 5, 6, 7); - __m128i values = lhs; - __m128i positions = indices; + const register_t indices = _mm_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7); + register_t values = lhs; + register_t positions = indices; auto reduce = [&]() noexcept { if constexpr (offset < 8) { - const __m128i shiftedValues = _mm_bsrli_si128(values, offset * 2); - const __m128i shiftedIndices = _mm_bsrli_si128(positions, offset * 2); - const __m128i less = _mm_cmpgt_epi16(values, shiftedValues); + const register_t shiftedValues = _mm_bsrli_si128(values, offset * 2); + const register_t shiftedIndices = _mm_bsrli_si128(positions, offset * 2); + const register_t less = _mm_cmpgt_epi16(values, shiftedValues); values = _mm_blendv_epi8(values, shiftedValues, less); positions = _mm_blendv_epi8(positions, shiftedIndices, less); } @@ -1026,118 +1043,118 @@ template <> struct SimdImpl128 return _mm_insert_epi16(values, _mm_extract_epi16(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm_abs_epi16(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi16(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_epi16(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_epi16(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm_slli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm_srli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm_srai_epi16(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm_hadd_epi16(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm_hsub_epi16(lhs, rhs); } /** @brief Horizontally adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept { return _mm_hadds_epi16(lhs, rhs); } /** @brief Horizontally subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept { return _mm_hsubs_epi16(lhs, rhs); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm_adds_epi16(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm_subs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept { - const __m128i lhsLo = _mm_cvtepi16_epi32(lhs); - const __m128i rhsLo = _mm_cvtepi16_epi32(rhs); - const __m128i lhsHi = _mm_cvtepi16_epi32(_mm_srli_si128(lhs, 8)); - const __m128i rhsHi = _mm_cvtepi16_epi32(_mm_srli_si128(rhs, 8)); + const register_t lhsLo = _mm_cvtepi16_epi32(lhs); + const register_t rhsLo = _mm_cvtepi16_epi32(rhs); + const register_t lhsHi = _mm_cvtepi16_epi32(_mm_srli_si128(lhs, 8)); + const register_t rhsHi = _mm_cvtepi16_epi32(_mm_srli_si128(rhs, 8)); return _mm_packs_epi32(_mm_mullo_epi32(lhsLo, rhsLo), _mm_mullo_epi32(lhsHi, rhsHi)); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi16(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm_set_epi16(static_cast(args)...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm_setr_epi16(static_cast(args)...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm_cmpgt_epi16(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtepi16_epi32(lhs, rhs); } @@ -1169,7 +1186,7 @@ template <> struct SimdImpl128 static_assert(dependent_false_v, "No direct widen mapping exists for SimdImpl128 and the requested destination SIMD shape."); } } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm_packs_epi16(lhs, rhs); } @@ -1185,7 +1202,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 8)`. * @return Selected scalar lane. */ - static int16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static int16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Signed 16-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -1211,12 +1228,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected signed 16-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int16_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int16_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 16-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int16_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int16_t rhs) noexcept { return _mm_insert_epi16(lhs, static_cast(rhs), index); } @@ -1227,20 +1244,20 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 8)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const int16_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int16_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Signed 16-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7); - const __m128i selected_lane = _mm_cmpeq_epi16(lane_indices, _mm_set1_epi16(static_cast(index))); + const register_t lane_indices = _mm_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7); + const register_t selected_lane = _mm_cmpeq_epi16(lane_indices, _mm_set1_epi16(static_cast(index))); return _mm_blendv_epi8(lhs, _mm_set1_epi16(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi16(lhs, rhs); } @@ -1251,12 +1268,12 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), false); } /** @brief Shuffles the low four 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(__m128i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(register_t lhs) noexcept { return _mm_shufflelo_epi16(lhs, imm8); } @@ -1265,12 +1282,12 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), true); } /** @brief Shuffles the high four 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(__m128i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(register_t lhs) noexcept { return _mm_shufflehi_epi16(lhs, imm8); } @@ -1280,23 +1297,30 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const register_t selected = _mm_and_si128(_mm_set1_epi16(static_cast(imm8)), selectors); + return _mm_blendv_epi8(lhs, rhs, _mm_cmpeq_epi16(selected, selectors)); } /** @brief Selects signed 16-bit lanes from two registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m128i lhs, const __m128i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm_blend_epi16(lhs, rhs, imm8); } }; template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects 16-bit lanes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -1309,37 +1333,37 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 8 && ((indices < 8) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi8(lhs, make_logical_shuffle_16_control(std::make_index_sequence<16>{})); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi16(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m128i lhsLo = _mm_cvtepu16_epi32(lhs); - const __m128i rhsLo = _mm_cvtepu16_epi32(rhs); - const __m128i lhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(lhs, 8)); - const __m128i rhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(rhs, 8)); + const register_t lhsLo = _mm_cvtepu16_epi32(lhs); + const register_t rhsLo = _mm_cvtepu16_epi32(rhs); + const register_t lhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(lhs, 8)); + const register_t rhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(rhs, 8)); return _mm_hadd_epi32(_mm_mullo_epi32(lhsLo, rhsLo), _mm_mullo_epi32(lhsHi, rhsHi)); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } /** @brief Computes the unchecked group magnitude in lane zero; all other lanes are unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - const __m128i lowProducts = _mm_mullo_epi16(lhs, lhs); - const __m128i highProducts = _mm_mulhi_epu16(lhs, lhs); - const __m128i lowSquares = _mm_unpacklo_epi16(lowProducts, highProducts); - const __m128i highSquares = _mm_unpackhi_epi16(lowProducts, highProducts); - __m128i total = _mm_add_epi32(lowSquares, highSquares); + const register_t lowProducts = _mm_mullo_epi16(lhs, lhs); + const register_t highProducts = _mm_mulhi_epu16(lhs, lhs); + const register_t lowSquares = _mm_unpacklo_epi16(lowProducts, highProducts); + const register_t highSquares = _mm_unpackhi_epi16(lowProducts, highProducts); + register_t total = _mm_add_epi32(lowSquares, highSquares); total = _mm_hadd_epi32(total, total); total = _mm_hadd_epi32(total, total); const std::uint32_t squareSum = static_cast(_mm_cvtsi128_si32(total)); @@ -1348,18 +1372,18 @@ template <> struct SimdImpl128 } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t threshold = maximum * maximum + maximum + 1; - const __m128i lowProducts = _mm_mullo_epi16(lhs, lhs); - const __m128i highProducts = _mm_mulhi_epu16(lhs, lhs); - const __m128i lowSquares = _mm_unpacklo_epi16(lowProducts, highProducts); - const __m128i highSquares = _mm_unpackhi_epi16(lowProducts, highProducts); - const __m128i lowPairs = _mm_add_epi64(_mm_cvtepu32_epi64(lowSquares), _mm_cvtepu32_epi64(_mm_srli_si128(lowSquares, 8))); - const __m128i highPairs = _mm_add_epi64(_mm_cvtepu32_epi64(highSquares), _mm_cvtepu32_epi64(_mm_srli_si128(highSquares, 8))); - const __m128i pairTotals = _mm_add_epi64(lowPairs, highPairs); - const __m128i totalVector = _mm_add_epi64(pairTotals, _mm_srli_si128(pairTotals, 8)); + const register_t lowProducts = _mm_mullo_epi16(lhs, lhs); + const register_t highProducts = _mm_mulhi_epu16(lhs, lhs); + const register_t lowSquares = _mm_unpacklo_epi16(lowProducts, highProducts); + const register_t highSquares = _mm_unpackhi_epi16(lowProducts, highProducts); + const register_t lowPairs = _mm_add_epi64(_mm_cvtepu32_epi64(lowSquares), _mm_cvtepu32_epi64(_mm_srli_si128(lowSquares, 8))); + const register_t highPairs = _mm_add_epi64(_mm_cvtepu32_epi64(highSquares), _mm_cvtepu32_epi64(_mm_srli_si128(highSquares, 8))); + const register_t pairTotals = _mm_add_epi64(lowPairs, highPairs); + const register_t totalVector = _mm_add_epi64(pairTotals, _mm_srli_si128(pairTotals, 8)); const std::uint64_t total = static_cast(_mm_cvtsi128_si64(totalVector)); const bool overflow = total >= threshold; const std::uint64_t result = overflow ? maximum : magnitude_round_sqrt_u64(total, maximum); @@ -1367,161 +1391,161 @@ template <> struct SimdImpl128 } /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { return _mm_minpos_epu16(lhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm_mullo_epi16(lhs, rhs); } /** @brief Divides corresponding unsigned 16-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epu16(lhs, rhs); } /** @brief Computes corresponding unsigned 16-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epu16(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { - const __m128i lo32 = _mm_cvtepu16_epi32(lhs); - const __m128i hi32 = _mm_cvtepu16_epi32(_mm_srli_si128(lhs, 8)); - const __m128i loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(lo32))); - const __m128i hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(hi32))); + const register_t lo32 = _mm_cvtepu16_epi32(lhs); + const register_t hi32 = _mm_cvtepu16_epi32(_mm_srli_si128(lhs, 8)); + const register_t loRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(lo32))); + const register_t hiRoots = _mm_cvtps_epi32(_mm_sqrt_ps(_ext_cvtepu32_ps(hi32))); return _mm_packus_epi32(loRoots, hiRoots); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm_abs_epi16(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi16(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_epu16(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_epu16(lhs, rhs); } /** @brief Computes lane-wise averages for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept { return _mm_avg_epu16(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm_slli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm_srli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm_srai_epi16(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm_hadd_epi16(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm_hsub_epi16(lhs, rhs); } /** @brief Horizontally adds unsigned 16-bit lanes with unsigned saturation. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept { - const __m128i zero = _mm_setzero_si128(); - const __m128i lhsPairs = _mm_adds_epu16(lhs, _mm_srli_epi32(lhs, 16)); - const __m128i rhsPairs = _mm_adds_epu16(rhs, _mm_srli_epi32(rhs, 16)); + const register_t zero = _mm_setzero_si128(); + const register_t lhsPairs = _mm_adds_epu16(lhs, _mm_srli_epi32(lhs, 16)); + const register_t rhsPairs = _mm_adds_epu16(rhs, _mm_srli_epi32(rhs, 16)); return _mm_packus_epi32(_mm_blend_epi16(lhsPairs, zero, 0xAA), _mm_blend_epi16(rhsPairs, zero, 0xAA)); } /** @brief Horizontally subtracts unsigned 16-bit lanes with unsigned saturation. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept { - const __m128i zero = _mm_setzero_si128(); - const __m128i lhsPairs = _mm_subs_epu16(lhs, _mm_srli_epi32(lhs, 16)); - const __m128i rhsPairs = _mm_subs_epu16(rhs, _mm_srli_epi32(rhs, 16)); + const register_t zero = _mm_setzero_si128(); + const register_t lhsPairs = _mm_subs_epu16(lhs, _mm_srli_epi32(lhs, 16)); + const register_t rhsPairs = _mm_subs_epu16(rhs, _mm_srli_epi32(rhs, 16)); return _mm_packus_epi32(_mm_blend_epi16(lhsPairs, zero, 0xAA), _mm_blend_epi16(rhsPairs, zero, 0xAA)); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm_adds_epu16(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm_subs_epu16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept { - const __m128i lhsLo = _mm_cvtepu16_epi32(lhs); - const __m128i rhsLo = _mm_cvtepu16_epi32(rhs); - const __m128i lhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(lhs, 8)); - const __m128i rhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(rhs, 8)); + const register_t lhsLo = _mm_cvtepu16_epi32(lhs); + const register_t rhsLo = _mm_cvtepu16_epi32(rhs); + const register_t lhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(lhs, 8)); + const register_t rhsHi = _mm_cvtepu16_epi32(_mm_srli_si128(rhs, 8)); return _mm_packus_epi32(_mm_mullo_epi32(lhsLo, rhsLo), _mm_mullo_epi32(lhsHi, rhsHi)); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi16(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm_set_epi16(static_cast(args)...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm_setr_epi16(static_cast(args)...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext_cmpgt_epu16(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtepu16_epi32(lhs, rhs); } @@ -1553,7 +1577,7 @@ template <> struct SimdImpl128 static_assert(dependent_false_v, "No direct widen mapping exists for SimdImpl128 and the requested destination SIMD shape."); } } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm_packus_epi16(lhs, rhs); } @@ -1569,7 +1593,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 8)`. * @return Selected scalar lane. */ - static uint16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static uint16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Unsigned 16-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -1595,12 +1619,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected unsigned 16-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint16_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint16_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 16-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint16_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint16_t rhs) noexcept { return _mm_insert_epi16(lhs, static_cast(rhs), index); } @@ -1611,20 +1635,20 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 8)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const uint16_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint16_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Unsigned 16-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7); - const __m128i selected_lane = _mm_cmpeq_epi16(lane_indices, _mm_set1_epi16(static_cast(index))); + const register_t lane_indices = _mm_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7); + const register_t selected_lane = _mm_cmpeq_epi16(lane_indices, _mm_set1_epi16(static_cast(index))); return _mm_blendv_epi8(lhs, _mm_set1_epi16(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi16(lhs, rhs); } @@ -1635,12 +1659,12 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), false); } /** @brief Shuffles the low four unsigned 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(__m128i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(register_t lhs) noexcept { return _mm_shufflelo_epi16(lhs, imm8); } @@ -1649,12 +1673,12 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), true); } /** @brief Shuffles the high four unsigned 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(__m128i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(register_t lhs) noexcept { return _mm_shufflehi_epi16(lhs, imm8); } @@ -1664,23 +1688,30 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const register_t selected = _mm_and_si128(_mm_set1_epi16(static_cast(imm8)), selectors); + return _mm_blendv_epi8(lhs, rhs, _mm_cmpeq_epi16(selected, selectors)); } /** @brief Selects unsigned 16-bit lanes from two registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m128i lhs, const __m128i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm_blend_epi16(lhs, rhs, imm8); } }; template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects 32-bit lanes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -1693,74 +1724,74 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 4 && ((indices < 4) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi32(lhs, encode_logical_shuffle_32_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi32(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m128i evenProducts = _mm_mul_epi32(lhs, rhs); - const __m128i oddProducts = _mm_mul_epi32(_mm_srli_si128(lhs, 4), _mm_srli_si128(rhs, 4)); + const register_t evenProducts = _mm_mul_epi32(lhs, rhs); + const register_t oddProducts = _mm_mul_epi32(_mm_srli_si128(lhs, 4), _mm_srli_si128(rhs, 4)); return _mm_add_epi64(evenProducts, oddProducts); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm_mullo_epi32(lhs, rhs); } /** @brief Divides corresponding signed 32-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epi32(lhs, rhs); } /** @brief Computes corresponding signed 32-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epi32(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128 roots = _mm_sqrt_ps(_mm_cvtepi32_ps(lhs)); return _mm_cvtps_epi32(roots); } /** @brief Computes the unchecked group magnitude in lane zero; all other lanes are unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - const __m128i pairSums = multiply_add_adjacent(lhs, lhs); - const __m128i totalVector = _mm_add_epi64(pairSums, _mm_srli_si128(pairSums, 8)); + const register_t pairSums = multiply_add_adjacent(lhs, lhs); + const register_t totalVector = _mm_add_epi64(pairSums, _mm_srli_si128(pairSums, 8)); const std::int64_t total = _mm_cvtsi128_si64(totalVector); const __m128d root = _mm_sqrt_sd(_mm_setzero_pd(), _mm_cvtsi64_sd(_mm_setzero_pd(), total)); return _mm_cvtsi64_si128(_mm_cvtsd_si64(root)); } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t threshold = maximum * maximum + maximum + 1; - const __m128i minimum = _mm_set1_epi32(std::numeric_limits::min()); + const register_t minimum = _mm_set1_epi32(std::numeric_limits::min()); if (_mm_movemask_epi8(_mm_cmpeq_epi32(lhs, minimum)) != 0) return magnitude_checked_result(maximum, true); - const __m128i evenSquares = _mm_mul_epi32(lhs, lhs); - const __m128i shifted = _mm_srli_si128(lhs, 4); - const __m128i oddSquares = _mm_mul_epi32(shifted, shifted); - const __m128i pairTotals = _mm_add_epi64(evenSquares, oddSquares); - const __m128i totalVector = _mm_add_epi64(pairTotals, _mm_srli_si128(pairTotals, 8)); + const register_t evenSquares = _mm_mul_epi32(lhs, lhs); + const register_t shifted = _mm_srli_si128(lhs, 4); + const register_t oddSquares = _mm_mul_epi32(shifted, shifted); + const register_t pairTotals = _mm_add_epi64(evenSquares, oddSquares); + const register_t totalVector = _mm_add_epi64(pairTotals, _mm_srli_si128(pairTotals, 8)); const std::uint64_t total = static_cast(_mm_cvtsi128_si64(totalVector)); const bool overflow = total >= threshold; const std::uint64_t result = overflow ? maximum : magnitude_round_sqrt_u64(total, maximum); @@ -1768,106 +1799,106 @@ template <> struct SimdImpl128 } /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept - { - constexpr __m128i indices = register_from_values<__m128i, std::int32_t>(0, 1, 2, 3); - __m128i values = lhs; - __m128i positions = indices; - const __m128i shifted1Values = _mm_bsrli_si128(values, 4); - const __m128i shifted1Indices = _mm_bsrli_si128(positions, 4); - const __m128i less1 = _mm_cmpgt_epi32(values, shifted1Values); + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + { + const register_t indices = _mm_setr_epi32(0, 1, 2, 3); + register_t values = lhs; + register_t positions = indices; + const register_t shifted1Values = _mm_bsrli_si128(values, 4); + const register_t shifted1Indices = _mm_bsrli_si128(positions, 4); + const register_t less1 = _mm_cmpgt_epi32(values, shifted1Values); values = _mm_blendv_epi8(values, shifted1Values, less1); positions = _mm_blendv_epi8(positions, shifted1Indices, less1); - const __m128i shifted2Values = _mm_bsrli_si128(values, 8); - const __m128i shifted2Indices = _mm_bsrli_si128(positions, 8); - const __m128i less2 = _mm_cmpgt_epi32(values, shifted2Values); + const register_t shifted2Values = _mm_bsrli_si128(values, 8); + const register_t shifted2Indices = _mm_bsrli_si128(positions, 8); + const register_t less2 = _mm_cmpgt_epi32(values, shifted2Values); values = _mm_blendv_epi8(values, shifted2Values, less2); positions = _mm_blendv_epi8(positions, shifted2Indices, less2); return _mm_insert_epi32(values, _mm_extract_epi32(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm_abs_epi32(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi32(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_epi32(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_epi32(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm_slli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm_srli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm_srai_epi32(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm_hadd_epi32(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm_hsub_epi32(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi32(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm_set_epi32(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm_setr_epi32(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm_cmpgt_epi32(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtepi32_epi64(lhs, rhs); } @@ -1889,7 +1920,7 @@ template <> struct SimdImpl128 static_assert(dependent_false_v, "No direct widen mapping exists for SimdImpl128 and the requested destination SIMD shape."); } } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm_packs_epi32(lhs, rhs); } @@ -1905,7 +1936,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 4)`. * @return Selected scalar lane. */ - static int32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static int32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Signed 32-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -1923,12 +1954,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected signed 32-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int32_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int32_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 32-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int32_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int32_t rhs) noexcept { return _mm_insert_epi32(lhs, rhs, index); } @@ -1939,20 +1970,20 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 4)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const int32_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int32_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Signed 32-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_setr_epi32(0, 1, 2, 3); - const __m128i selected_lane = _mm_cmpeq_epi32(lane_indices, _mm_set1_epi32(index)); + const register_t lane_indices = _mm_setr_epi32(0, 1, 2, 3); + const register_t selected_lane = _mm_cmpeq_epi32(lane_indices, _mm_set1_epi32(index)); return _mm_blendv_epi8(lhs, _mm_set1_epi32(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi32(lhs, rhs); } @@ -1963,7 +1994,7 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), false); } @@ -1972,7 +2003,7 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), true); } @@ -1982,23 +2013,30 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const register_t selected = _mm_and_si128(_mm_set1_epi32(imm8), selectors); + return _mm_blendv_epi8(lhs, rhs, _mm_cmpeq_epi32(selected, selectors)); } /** @brief Selects signed 32-bit lanes from two registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m128i lhs, const __m128i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm_castps_si128(_mm_blend_ps(_mm_castsi128_ps(lhs), _mm_castsi128_ps(rhs), imm8 & 0x0F)); } }; template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects 32-bit lanes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -2011,12 +2049,12 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 4 && ((indices < 4) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi32(lhs, encode_logical_shuffle_32_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi32(lhs, rhs); } @@ -2026,27 +2064,27 @@ template <> struct SimdImpl128 * @param lhs The unsigned integer lanes. * @return The converted floating-point lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) convert_to_float(const __m128i lhs) noexcept + static auto SIMD_FLAGS(InOut, ForceInline) convert_to_float(const register_t lhs) noexcept { return _ext_cvtepu32_ps(lhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m128i evenProducts = _mm_mul_epu32(lhs, rhs); - const __m128i oddProducts = _mm_mul_epu32(_mm_srli_si128(lhs, 4), _mm_srli_si128(rhs, 4)); + const register_t evenProducts = _mm_mul_epu32(lhs, rhs); + const register_t oddProducts = _mm_mul_epu32(_mm_srli_si128(lhs, 4), _mm_srli_si128(rhs, 4)); return _mm_add_epi64(evenProducts, oddProducts); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm_mullo_epi32(lhs, rhs); } @@ -2057,42 +2095,42 @@ template <> struct SimdImpl128 * @param rhs The nonzero divisor lanes. * @return The truncating integer quotients. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epu32(lhs, rhs); } /** @brief Computes corresponding unsigned 32-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epu32(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128 roots = _mm_sqrt_ps(_ext_cvtepu32_ps(lhs)); return _mm_cvtps_epi32(roots); } /** @brief Computes the unchecked group magnitude in lane zero; all other lanes are unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - const __m128i pairSums = multiply_add_adjacent(lhs, lhs); - const __m128i totalVector = _mm_add_epi64(pairSums, _mm_srli_si128(pairSums, 8)); + const register_t pairSums = multiply_add_adjacent(lhs, lhs); + const register_t totalVector = _mm_add_epi64(pairSums, _mm_srli_si128(pairSums, 8)); const std::uint64_t total = static_cast(_mm_cvtsi128_si64(totalVector)); const __m128d root = _mm_sqrt_sd(_mm_setzero_pd(), _mm_set_sd(static_cast(total))); return _mm_cvtsi64_si128(_mm_cvtsd_si64(root)); } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t threshold = maximum * maximum + maximum + 1; - const __m128i evenSquares = _mm_mul_epu32(lhs, lhs); - const __m128i shifted = _mm_srli_si128(lhs, 4); - const __m128i oddSquares = _mm_mul_epu32(shifted, shifted); - const __m128i pairTotals = _mm_add_epi64(evenSquares, oddSquares); - const __m128i signBit = _mm_set1_epi64x(std::numeric_limits::min()); - const __m128i pairCarries = _mm_cmpgt_epi64(_mm_xor_si128(evenSquares, signBit), _mm_xor_si128(pairTotals, signBit)); + const register_t evenSquares = _mm_mul_epu32(lhs, lhs); + const register_t shifted = _mm_srli_si128(lhs, 4); + const register_t oddSquares = _mm_mul_epu32(shifted, shifted); + const register_t pairTotals = _mm_add_epi64(evenSquares, oddSquares); + const register_t signBit = _mm_set1_epi64x(std::numeric_limits::min()); + const register_t pairCarries = _mm_cmpgt_epi64(_mm_xor_si128(evenSquares, signBit), _mm_xor_si128(pairTotals, signBit)); const std::uint64_t lowTotal = static_cast(_mm_cvtsi128_si64(pairTotals)); const std::uint64_t highTotal = static_cast(_mm_extract_epi64(pairTotals, 1)); const std::uint64_t total = lowTotal + highTotal; @@ -2101,108 +2139,119 @@ template <> struct SimdImpl128 return magnitude_checked_result(result, overflow); } + /** @brief Returns the unsigned-doubleword lane indices as one compile-time-generated native register. */ + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) lane_indices() noexcept + { + return make_static_register>([]() constexpr noexcept + { + return static_cast(index); + }); + } + /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept - { - constexpr __m128i indices = register_from_values<__m128i, std::uint32_t>(0u, 1u, 2u, 3u); - const __m128i signBit = _mm_set1_epi32(static_cast(0x80000000u)); - __m128i values = lhs; - __m128i positions = indices; - const __m128i shifted1Values = _mm_bsrli_si128(values, 4); - const __m128i shifted1Indices = _mm_bsrli_si128(positions, 4); - const __m128i less1 = _mm_cmpgt_epi32(_mm_xor_si128(values, signBit), _mm_xor_si128(shifted1Values, signBit)); + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + { + const register_t indices = lane_indices(); + const register_t signBit = _mm_set1_epi32(static_cast(0x80000000u)); + register_t values = lhs; + register_t positions = indices; + const register_t shifted1Values = _mm_bsrli_si128(values, 4); + const register_t shifted1Indices = _mm_bsrli_si128(positions, 4); + const register_t less1 = _mm_cmpgt_epi32(_mm_xor_si128(values, signBit), _mm_xor_si128(shifted1Values, signBit)); values = _mm_blendv_epi8(values, shifted1Values, less1); positions = _mm_blendv_epi8(positions, shifted1Indices, less1); - const __m128i shifted2Values = _mm_bsrli_si128(values, 8); - const __m128i shifted2Indices = _mm_bsrli_si128(positions, 8); - const __m128i less2 = _mm_cmpgt_epi32(_mm_xor_si128(values, signBit), _mm_xor_si128(shifted2Values, signBit)); + const register_t shifted2Values = _mm_bsrli_si128(values, 8); + const register_t shifted2Indices = _mm_bsrli_si128(positions, 8); + const register_t less2 = _mm_cmpgt_epi32(_mm_xor_si128(values, signBit), _mm_xor_si128(shifted2Values, signBit)); values = _mm_blendv_epi8(values, shifted2Values, less2); positions = _mm_blendv_epi8(positions, shifted2Indices, less2); return _mm_insert_epi32(values, _mm_extract_epi32(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm_abs_epi32(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi32(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_epu32(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_epu32(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm_slli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm_srli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm_srai_epi32(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm_hadd_epi32(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm_hsub_epi32(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + /** @brief Broadcasts one value to all unsigned 32-bit lanes. */ + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi32(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm_set_epi32(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + /** @brief Constructs unsigned 32-bit lanes in low-to-high logical order. */ + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm_setr_epi32(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext_cmpgt_epu32(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtepu32_epi64(lhs, rhs); } @@ -2224,7 +2273,7 @@ template <> struct SimdImpl128 static_assert(dependent_false_v, "No direct widen mapping exists for SimdImpl128 and the requested destination SIMD shape."); } } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm_packus_epi32(lhs, rhs); } @@ -2240,7 +2289,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 4)`. * @return Selected scalar lane. */ - static uint32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static uint32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Unsigned 32-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -2258,12 +2307,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected unsigned 32-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint32_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint32_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 32-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint32_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint32_t rhs) noexcept { return _mm_insert_epi32(lhs, std::bit_cast(rhs), index); } @@ -2274,20 +2323,20 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 4)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const uint32_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint32_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Unsigned 32-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_setr_epi32(0, 1, 2, 3); - const __m128i selected_lane = _mm_cmpeq_epi32(lane_indices, _mm_set1_epi32(index)); + const register_t lane_indices = _mm_setr_epi32(0, 1, 2, 3); + const register_t selected_lane = _mm_cmpeq_epi32(lane_indices, _mm_set1_epi32(index)); return _mm_blendv_epi8(lhs, _mm_set1_epi32(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi32(lhs, rhs); } @@ -2298,7 +2347,7 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), false); } @@ -2307,7 +2356,7 @@ template <> struct SimdImpl128 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), true); } @@ -2317,23 +2366,30 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const register_t selected = _mm_and_si128(_mm_set1_epi32(imm8), selectors); + return _mm_blendv_epi8(lhs, rhs, _mm_cmpeq_epi32(selected, selectors)); } /** @brief Selects unsigned 32-bit lanes from two registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m128i lhs, const __m128i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm_castps_si128(_mm_blend_ps(_mm_castsi128_ps(lhs), _mm_castsi128_ps(rhs), imm8 & 0x0F)); } }; template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects 64-bit lanes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -2346,52 +2402,52 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 2 && ((indices < 2) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi32(lhs, encode_logical_shuffle_64_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi64(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept - { - const __m128i productLow = _mm_mul_epu32(lhs, rhs); - const __m128i lhsHigh = _mm_srli_epi64(lhs, 32); - const __m128i rhsHigh = _mm_srli_epi64(rhs, 32); - const __m128i cross = _mm_add_epi64(_mm_mul_epu32(lhsHigh, rhs), _mm_mul_epu32(lhs, rhsHigh)); - const __m128i products = _mm_add_epi64(productLow, _mm_slli_epi64(cross, 32)); - const __m128i shifted = _mm_bsrli_si128(products, 8); - const __m128i sum = _mm_add_epi64(products, shifted); + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + { + const register_t productLow = _mm_mul_epu32(lhs, rhs); + const register_t lhsHigh = _mm_srli_epi64(lhs, 32); + const register_t rhsHigh = _mm_srli_epi64(rhs, 32); + const register_t cross = _mm_add_epi64(_mm_mul_epu32(lhsHigh, rhs), _mm_mul_epu32(lhs, rhsHigh)); + const register_t products = _mm_add_epi64(productLow, _mm_slli_epi64(cross, 32)); + const register_t shifted = _mm_bsrli_si128(products, 8); + const register_t sum = _mm_add_epi64(products, shifted); return _mm_unpacklo_epi64(sum, _mm_setzero_si128()); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext_mullo_epi64(lhs, rhs); } /** @brief Divides corresponding signed 64-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epi64(lhs, rhs); } /** @brief Computes corresponding signed 64-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epi64(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128d roots = _mm_sqrt_pd(_mm_setr_pd(static_cast(_mm_cvtsi128_si64(lhs)), static_cast(_mm_extract_epi64(lhs, 1)))); const auto lowRoot = static_cast(_mm_cvtsd_f64(roots)); @@ -2399,7 +2455,7 @@ template <> struct SimdImpl128 return _mm_set_epi64x(highRoot, lowRoot); } /** @brief Computes the unchecked group magnitude in lane zero; lane one is unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const std::uint64_t rawLow = static_cast(_mm_cvtsi128_si64(lhs)); const std::uint64_t rawHigh = static_cast(_mm_extract_epi64(lhs, 1)); @@ -2407,8 +2463,8 @@ template <> struct SimdImpl128 const std::uint64_t highSign = rawHigh >> 63; const std::uint64_t lowValue = (rawLow ^ (std::uint64_t{0} - lowSign)) + lowSign; const std::uint64_t highValue = (rawHigh ^ (std::uint64_t{0} - highSign)) + highSign; - const __m128i lowSquare = magnitude_square_u64(lowValue); - const __m128i highSquare = magnitude_square_u64(highValue); + const register_t lowSquare = magnitude_square_u64(lowValue); + const register_t highSquare = magnitude_square_u64(highValue); const std::uint64_t lowWord0 = static_cast(_mm_cvtsi128_si64(lowSquare)); const std::uint64_t lowWord1 = static_cast(_mm_cvtsi128_si64(highSquare)); const std::uint64_t highWord0 = static_cast(_mm_extract_epi64(lowSquare, 1)); @@ -2420,7 +2476,7 @@ template <> struct SimdImpl128 } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = static_cast(std::numeric_limits::max()); constexpr std::uint64_t thresholdLow = 0x8000'0000'0000'0001ULL; @@ -2431,8 +2487,8 @@ template <> struct SimdImpl128 const std::uint64_t highSign = rawHigh >> 63; const std::uint64_t lowValue = (rawLow ^ (std::uint64_t{0} - lowSign)) + lowSign; const std::uint64_t highValue = (rawHigh ^ (std::uint64_t{0} - highSign)) + highSign; - const __m128i lowSquare = magnitude_square_u64(lowValue); - const __m128i highSquare = magnitude_square_u64(highValue); + const register_t lowSquare = magnitude_square_u64(lowValue); + const register_t highSquare = magnitude_square_u64(highValue); const std::uint64_t lowWord0 = static_cast(_mm_cvtsi128_si64(lowSquare)); const std::uint64_t lowWord1 = static_cast(_mm_cvtsi128_si64(highSquare)); const std::uint64_t highWord0 = static_cast(_mm_extract_epi64(lowSquare, 1)); @@ -2448,67 +2504,67 @@ template <> struct SimdImpl128 } /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept - { - constexpr __m128i indices = register_from_values<__m128i, std::int64_t>(0, 1); - const __m128i shiftedValues = _mm_bsrli_si128(lhs, 8); - const __m128i shiftedIndices = _mm_bsrli_si128(indices, 8); - const __m128i less = _mm_cmpgt_epi64(lhs, shiftedValues); - const __m128i values = _mm_blendv_epi8(lhs, shiftedValues, less); - const __m128i positions = _mm_blendv_epi8(indices, shiftedIndices, less); + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + { + const register_t indices = _mm_set_epi64x(1, 0); + const register_t shiftedValues = _mm_bsrli_si128(lhs, 8); + const register_t shiftedIndices = _mm_bsrli_si128(indices, 8); + const register_t less = _mm_cmpgt_epi64(lhs, shiftedValues); + const register_t values = _mm_blendv_epi8(lhs, shiftedValues, less); + const register_t positions = _mm_blendv_epi8(indices, shiftedIndices, less); return _mm_insert_epi64(values, _mm_extract_epi64(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _ext_abs_epi64(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi64(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _ext_min_epi64(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _ext_max_epi64(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm_slli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm_srli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext_srai_epi64(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi64x(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm_set_epi64x(args...); } @@ -2518,17 +2574,17 @@ template <> struct SimdImpl128 * @param high Value for lane one. * @return Register containing low followed by high. */ - static __m128i SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(const std::int64_t low, const std::int64_t high) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(const std::int64_t low, const std::int64_t high) noexcept { return _mm_set_epi64x(high, low); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm_cmpgt_epi64(lhs, rhs); } @@ -2544,7 +2600,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 2)`. * @return Selected scalar lane. */ - static int64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static int64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 2, "Signed 64-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -2558,12 +2614,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected signed 64-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int64_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int64_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 64-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int64_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int64_t rhs) noexcept { return _mm_insert_epi64(lhs, rhs, index); } @@ -2574,20 +2630,20 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 2)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const int64_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int64_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 2, "Signed 64-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_set_epi64x(1, 0); - const __m128i selected_lane = _mm_cmpeq_epi64(lane_indices, _mm_set1_epi64x(index)); + const register_t lane_indices = _mm_set_epi64x(1, 0); + const register_t selected_lane = _mm_cmpeq_epi64(lane_indices, _mm_set1_epi64x(index)); return _mm_blendv_epi8(lhs, _mm_set1_epi64x(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi64(lhs, rhs); } @@ -2595,8 +2651,10 @@ template <> struct SimdImpl128 template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128i; /** @brief Selects 64-bit lanes from two registers using a canonical predicate register. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128i condition, __m128i when_true, __m128i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_epi8(when_false, when_true, condition); } @@ -2609,52 +2667,52 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 2 && ((indices < 2) && ...)) - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_epi32(lhs, encode_logical_shuffle_64_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_epi64(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept - { - const __m128i productLow = _mm_mul_epu32(lhs, rhs); - const __m128i lhsHigh = _mm_srli_epi64(lhs, 32); - const __m128i rhsHigh = _mm_srli_epi64(rhs, 32); - const __m128i cross = _mm_add_epi64(_mm_mul_epu32(lhsHigh, rhs), _mm_mul_epu32(lhs, rhsHigh)); - const __m128i products = _mm_add_epi64(productLow, _mm_slli_epi64(cross, 32)); - const __m128i shifted = _mm_bsrli_si128(products, 8); - const __m128i sum = _mm_add_epi64(products, shifted); + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + { + const register_t productLow = _mm_mul_epu32(lhs, rhs); + const register_t lhsHigh = _mm_srli_epi64(lhs, 32); + const register_t rhsHigh = _mm_srli_epi64(rhs, 32); + const register_t cross = _mm_add_epi64(_mm_mul_epu32(lhsHigh, rhs), _mm_mul_epu32(lhs, rhsHigh)); + const register_t products = _mm_add_epi64(productLow, _mm_slli_epi64(cross, 32)); + const register_t shifted = _mm_bsrli_si128(products, 8); + const register_t sum = _mm_add_epi64(products, shifted); return _mm_unpacklo_epi64(sum, _mm_setzero_si128()); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext_mullo_epi64(lhs, rhs); } /** @brief Divides corresponding unsigned 64-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext128_div_epu64(lhs, rhs); } /** @brief Computes corresponding unsigned 64-bit remainders with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext128_rem_epu64(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128d roots = _mm_sqrt_pd(_mm_setr_pd(static_cast(static_cast(_mm_cvtsi128_si64(lhs))), static_cast(static_cast(_mm_extract_epi64(lhs, 1))))); @@ -2663,12 +2721,12 @@ template <> struct SimdImpl128 return _mm_set_epi64x(highRoot, lowRoot); } /** @brief Computes the unchecked group magnitude in lane zero; lane one is unspecified. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const std::uint64_t lowValue = static_cast(_mm_cvtsi128_si64(lhs)); const std::uint64_t highValue = static_cast(_mm_extract_epi64(lhs, 1)); - const __m128i lowSquare = magnitude_square_u64(lowValue); - const __m128i highSquare = magnitude_square_u64(highValue); + const register_t lowSquare = magnitude_square_u64(lowValue); + const register_t highSquare = magnitude_square_u64(highValue); const std::uint64_t lowWord0 = static_cast(_mm_cvtsi128_si64(lowSquare)); const std::uint64_t lowWord1 = static_cast(_mm_cvtsi128_si64(highSquare)); const std::uint64_t highWord0 = static_cast(_mm_extract_epi64(lowSquare, 1)); @@ -2680,15 +2738,15 @@ template <> struct SimdImpl128 } /** @brief Computes a saturated magnitude in lane zero and a canonical overflow mask in lane one. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { constexpr std::uint64_t maximum = std::numeric_limits::max(); constexpr std::uint64_t thresholdLow = 1; constexpr std::uint64_t thresholdHigh = maximum; const std::uint64_t lowValue = static_cast(_mm_cvtsi128_si64(lhs)); const std::uint64_t highValue = static_cast(_mm_extract_epi64(lhs, 1)); - const __m128i lowSquare = magnitude_square_u64(lowValue); - const __m128i highSquare = magnitude_square_u64(highValue); + const register_t lowSquare = magnitude_square_u64(lowValue); + const register_t highSquare = magnitude_square_u64(highValue); const std::uint64_t lowWord0 = static_cast(_mm_cvtsi128_si64(lowSquare)); const std::uint64_t lowWord1 = static_cast(_mm_cvtsi128_si64(highSquare)); const std::uint64_t highWord0 = static_cast(_mm_extract_epi64(lowSquare, 1)); @@ -2703,69 +2761,79 @@ template <> struct SimdImpl128 return magnitude_checked_result(result, overflow); } + /** @brief Returns the unsigned-quadword lane indices as one compile-time-generated native register. */ + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) lane_indices() noexcept + { + return make_static_register>([]() constexpr noexcept + { + return static_cast(index); + }); + } + /** @brief Computes minimum-value position metadata for this native register specialization. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept - { - constexpr __m128i indices = register_from_values<__m128i, std::uint64_t>(0ull, 1ull); - const __m128i signBit = _mm_set1_epi64x(std::numeric_limits::min()); - const __m128i shiftedValues = _mm_bsrli_si128(lhs, 8); - const __m128i shiftedIndices = _mm_bsrli_si128(indices, 8); - const __m128i less = _mm_cmpgt_epi64(_mm_xor_si128(lhs, signBit), _mm_xor_si128(shiftedValues, signBit)); - const __m128i values = _mm_blendv_epi8(lhs, shiftedValues, less); - const __m128i positions = _mm_blendv_epi8(indices, shiftedIndices, less); + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + { + const register_t indices = lane_indices(); + const register_t signBit = _mm_set1_epi64x(std::numeric_limits::min()); + const register_t shiftedValues = _mm_bsrli_si128(lhs, 8); + const register_t shiftedIndices = _mm_bsrli_si128(indices, 8); + const register_t less = _mm_cmpgt_epi64(_mm_xor_si128(lhs, signBit), _mm_xor_si128(shiftedValues, signBit)); + const register_t values = _mm_blendv_epi8(lhs, shiftedValues, less); + const register_t positions = _mm_blendv_epi8(indices, shiftedIndices, less); return _mm_insert_epi64(values, _mm_extract_epi64(positions, 0), 1); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m128i lhs, __m128i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return lhs; } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm_sub_epi64(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _ext_min_epu64(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _ext_max_epu64(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm_slli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm_srli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext_srai_epi64(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + /** @brief Broadcasts one value to all unsigned 64-bit lanes. */ + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_epi64x(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm_set_epi64x(args...); } @@ -2775,17 +2843,17 @@ template <> struct SimdImpl128 * @param high Value for lane one. * @return Register containing the exact low and high lane bit patterns. */ - static __m128i SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(const std::uint64_t low, const std::uint64_t high) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(const std::uint64_t low, const std::uint64_t high) noexcept { return _mm_set_epi64x(std::bit_cast(high), std::bit_cast(low)); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext_cmpgt_epu64(lhs, rhs); } @@ -2801,7 +2869,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 2)`. * @return Selected scalar lane. */ - static uint64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128i lhs, const int index) noexcept + static uint64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 2, "Unsigned 64-bit extraction requires a valid 128-bit lane index"); switch (index) @@ -2815,12 +2883,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected unsigned 64-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint64_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint64_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 64-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint64_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint64_t rhs) noexcept { return _mm_insert_epi64(lhs, std::bit_cast(rhs), index); } @@ -2831,20 +2899,20 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 2)`. * @return Register with the selected lane replaced. */ - static __m128i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128i lhs, const uint64_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint64_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 2, "Unsigned 64-bit insertion requires a valid 128-bit lane index"); - const __m128i lane_indices = _mm_set_epi64x(1, 0); - const __m128i selected_lane = _mm_cmpeq_epi64(lane_indices, _mm_set1_epi64x(index)); + const register_t lane_indices = _mm_set_epi64x(1, 0); + const register_t selected_lane = _mm_cmpeq_epi64(lane_indices, _mm_set1_epi64x(index)); return _mm_blendv_epi8(lhs, _mm_set1_epi64x(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_epi64(lhs, rhs); } @@ -2852,8 +2920,10 @@ template <> struct SimdImpl128 template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128; /** @brief Selects float lanes from two registers using a canonical predicate register. */ - static __m128 SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128 condition, __m128 when_true, __m128 when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_ps(when_false, when_true, condition); } @@ -2866,44 +2936,44 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 4 && ((indices < 4) && ...)) - static __m128 SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128 lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_ps(lhs, lhs, encode_logical_shuffle_32_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_ps(lhs, rhs); } /** @brief Alternates lane subtraction and addition for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept { return _mm_addsub_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm_mul_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _mm_div_ps(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { return _mm_sqrt_ps(lhs); } /** @brief Computes and broadcasts the 128-bit floating-point magnitude. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { return _mm_sqrt_ps(_mm_dp_ps(lhs, lhs, 0xFF)); } /** @brief Multiplies lanes and adds a third register for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept { #if SIMDLIB_HAS_FMA return _mm_fmadd_ps(lhs, rhs, addend); @@ -2912,69 +2982,69 @@ template <> struct SimdImpl128 #endif } /** @brief Computes an immediate-controlled dot product for this native register specialization. */ - template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(__m128 lhs, __m128 rhs) noexcept + template static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(register_t lhs, register_t rhs) noexcept { return _mm_dp_ps(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _ext_abs_ps(lhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_ps(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_ps(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm_hadd_ps(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm_hsub_ps(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set_ps1(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm_set_ps(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm_setr_ps(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm_cmpgt_ps(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtps_epi32(lhs, rhs); } - // static auto SIMD_FLAGS(InOut, ForceInline) compress (auto lhs, auto rhs) noexcept { return _mm_cvtepi32_ps(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) compress (auto lhs, auto rhs) noexcept { return _mm_cvtepi32_ps(lhs, rhs); } // extract / insert template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline) extract(auto lhs) noexcept @@ -2988,7 +3058,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 4)`. * @return Selected scalar lane. */ - static float SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128 lhs, const int index) noexcept + static float SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "32-bit floating-point extraction requires a valid 128-bit lane index"); switch (index) @@ -3006,12 +3076,12 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected 32-bit floating-point lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const float rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const float rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected 32-bit floating-point lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const float rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const float rhs) noexcept { return _mm_insert_ps(lhs, _mm_set_ss(rhs), index << 4); } @@ -3022,7 +3092,7 @@ template <> struct SimdImpl128 * @param index Selected lane index. * @return Register with the selected lane replaced. */ - static __m128 SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128 lhs, const float rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const float rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "32-bit floating-point insertion requires a valid 128-bit lane index"); switch (index) @@ -3039,11 +3109,11 @@ template <> struct SimdImpl128 } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_ps(lhs, rhs); } @@ -3055,7 +3125,7 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the shuffled lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, unsigned int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, unsigned int imm8) noexcept { return register_shuffle_float_slow(lhs, rhs, imm8); } @@ -3065,18 +3135,20 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - const unsigned int control = static_cast(imm8); - const __m128 mask = _mm_castsi128_ps(_mm_set_epi32(-static_cast((control >> 3) & 0x1u), -static_cast((control >> 2) & 0x1u), - -static_cast((control >> 1) & 0x1u), -static_cast(control & 0x1u))); - return _mm_blendv_ps(lhs, rhs, mask); + const __m128i selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const __m128i selected = _mm_and_si128(_mm_set1_epi32(imm8), selectors); + return _mm_blendv_ps(lhs, rhs, _mm_castsi128_ps(_mm_cmpeq_epi32(selected, selectors))); } /** @brief Selects 32-bit floating-point lanes from two registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m128 lhs, const __m128 rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm_blend_ps(lhs, rhs, imm8 & 0x0F); } static auto SIMD_FLAGS(In, RegisterOnly, ForceInline) movemask(auto lhs) noexcept @@ -3087,8 +3159,10 @@ template <> struct SimdImpl128 template <> struct SimdImpl128 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m128d; /** @brief Selects double lanes from two registers using a canonical predicate register. */ - static __m128d SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m128d condition, __m128d when_true, __m128d when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm_blendv_pd(when_false, when_true, condition); } @@ -3101,44 +3175,44 @@ template <> struct SimdImpl128 */ template requires(sizeof...(indices) == 2 && ((indices < 2) && ...)) - static __m128d SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m128d lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm_shuffle_pd(lhs, lhs, encode_logical_shuffle_double_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm_add_pd(lhs, rhs); } /** @brief Alternates lane subtraction and addition for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept { return _mm_addsub_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm_sub_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm_mul_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _mm_div_pd(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { return _mm_sqrt_pd(lhs); } /** @brief Computes and broadcasts the 128-bit floating-point magnitude. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { return _mm_sqrt_pd(_mm_dp_pd(lhs, lhs, 0x33)); } /** @brief Multiplies lanes and adds a third register for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept { #if SIMDLIB_HAS_FMA return _mm_fmadd_pd(lhs, rhs, addend); @@ -3147,70 +3221,70 @@ template <> struct SimdImpl128 #endif } /** @brief Computes an immediate-controlled dot product for this native register specialization. */ - template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(__m128d lhs, __m128d rhs) noexcept + template static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(register_t lhs, register_t rhs) noexcept { return _mm_dp_pd(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _ext_abs_pd(lhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm_min_pd(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm_max_pd(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm_hadd_pd(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm_hsub_pd(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm_set1_pd(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm_set_pd(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm_setr_pd(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm_cmpeq_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm_cmpgt_pd(lhs, rhs); } - // static auto SIMD_FLAGS(InOut, ForceInline) cmplt (auto lhs, auto rhs) noexcept { return _mm_cmplt_pd(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) cmplt (auto lhs, auto rhs) noexcept { return _mm_cmplt_pd(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm_cvtps_epi32(lhs, rhs); } - // static auto SIMD_FLAGS(InOut, ForceInline) compress (auto lhs, auto rhs) noexcept { return _mm_cvtepi32_pd(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) compress (auto lhs, auto rhs) noexcept { return _mm_cvtepi32_pd(lhs, rhs); } // extract / insert template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline) extract(auto lhs) noexcept @@ -3226,7 +3300,7 @@ template <> struct SimdImpl128 * @param index Selected lane in the range `[0, 2)`. * @return Selected scalar lane. */ - static double SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m128d lhs, const int index) noexcept + static double SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 2, "64-bit floating-point extraction requires a valid 128-bit lane index"); switch (index) @@ -3240,14 +3314,14 @@ template <> struct SimdImpl128 } } /** @brief Replaces the compile-time-selected 64-bit floating-point lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const double rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const double rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected 64-bit floating-point lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const double rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const double rhs) noexcept { - const __m128d replacement = _mm_set_sd(rhs); + const register_t replacement = _mm_set_sd(rhs); if constexpr (index == 0) return _mm_move_sd(lhs, replacement); else @@ -3260,7 +3334,7 @@ template <> struct SimdImpl128 * @param index Selected lane index. * @return Register with the selected lane replaced. */ - static __m128d SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m128d lhs, const double rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const double rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 2, "64-bit floating-point insertion requires a valid 128-bit lane index"); switch (index) @@ -3273,11 +3347,11 @@ template <> struct SimdImpl128 } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm_unpacklo_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm_unpackhi_pd(lhs, rhs); } @@ -3289,7 +3363,7 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the shuffled lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, unsigned int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, unsigned int imm8) noexcept { return register_shuffle_double_slow(lhs, rhs, imm8); } @@ -3299,17 +3373,20 @@ template <> struct SimdImpl128 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - const unsigned int control = static_cast(imm8); - const __m128d mask = _mm_castsi128_pd(_mm_set_epi64x(-static_cast((control >> 1) & 0x1u), -static_cast(control & 0x1u))); - return _mm_blendv_pd(lhs, rhs, mask); + const __m128i selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const __m128i selected = _mm_and_si128(_mm_set1_epi64x(imm8), selectors); + return _mm_blendv_pd(lhs, rhs, _mm_castsi128_pd(_mm_cmpeq_epi64(selected, selectors))); } /** @brief Selects 64-bit floating-point lanes from two registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m128d lhs, const __m128d rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm_blend_pd(lhs, rhs, imm8 & 0x03); } static auto SIMD_FLAGS(In, RegisterOnly, ForceInline) movemask(auto lhs) noexcept @@ -3384,7 +3461,7 @@ template struct SimdMappings<128, element_t> : public SimdImpl { if (std::is_constant_evaluated()) { - return register_from_array(data); + return register_from_array_constexpr(data); } else { @@ -3819,7 +3896,22 @@ template struct SimdMappings<128, element_t> : public SimdImpl /// Returns a mask of the most significant BIT of each element. static mask_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) movemask_slim(const vector_t lhs) noexcept { - if constexpr (std::is_integral_v) + if constexpr (std::is_integral_v && sizeof(element_t) == 1) + { + // Byte lanes already match MOVMSK's native granularity, so no lane compaction is required. + return _mm_movemask_epi8(lhs); + } + else if constexpr (std::is_integral_v && sizeof(element_t) == sizeof(float)) + { + // Reinterpret each integer lane as float so MOVMSKPS reads the same lane sign bits without conversion. + return _mm_movemask_ps(_mm_castsi128_ps(lhs)); + } + else if constexpr (std::is_integral_v && sizeof(element_t) == sizeof(double)) + { + // Reinterpret each integer lane as double so MOVMSKPD reads the same lane sign bits without conversion. + return _mm_movemask_pd(_mm_castsi128_pd(lhs)); + } + else if constexpr (std::is_integral_v) return movemask(swizzle_msb(lhs)); else if constexpr (std::is_same_v) return _mm_movemask_ps(lhs); @@ -3850,26 +3942,17 @@ template struct SimdMappings<128, element_t> : public SimdImpl } /// Returns the shuffle order to move the most significant byte of each element into the least significant bytes. - constexpr static int_vector_t get_msb_swizzle_order() noexcept + constexpr static int_vector_t SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) get_msb_swizzle_order() noexcept { - int_vector_t seq{}; - constexpr const auto elem_size = sizeof(element_t); - constexpr const auto elem_count = 16 / elem_size; - - for (std::size_t i = 0; i < 16; ++i) + return make_static_register>([]() constexpr noexcept { - if (i < elem_count) - { - // Select the MSB byte of each element, packing them into the low bytes. - register_set_constexpr(seq, i, static_cast((i * elem_size) + (elem_size - 1))); - } + constexpr std::size_t element_size = sizeof(element_t); + constexpr std::size_t element_count = 16 / element_size; + if constexpr (index < element_count) + return static_cast((index * element_size) + (element_size - 1)); else - { - // Zero out the rest (PSHUFB: high bit set => 0). - register_set_constexpr(seq, i, 0x80); - } - } - return seq; + return std::uint8_t{0x80}; + }); } /// Swizzle the vector to only contain the most significant bit of each byte. @@ -3957,8 +4040,10 @@ static __m256i SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) make_logical_ template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects bytes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -3971,7 +4056,7 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 32 && ((indices < 32) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { constexpr auto selectors = std::array{indices...}; if constexpr (!logical_shuffle_256_has_cross_half_selector<16, selectors>()) @@ -3980,7 +4065,7 @@ template <> struct SimdImpl256 } else { - const __m256i swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); + const register_t swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); if constexpr (!logical_shuffle_256_has_local_half_selector<16, selectors>()) return _mm256_shuffle_epi8(swapped, make_logical_shuffle_256_byte_control<1, true, selectors>(std::make_index_sequence<32>{})); else @@ -3990,53 +4075,53 @@ template <> struct SimdImpl256 } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi8(lhs, rhs); } /** @brief Multiplies signed byte lanes and adds adjacent products into signed 16-bit lanes. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m256i lowProducts = _mm256_mullo_epi16(_mm256_cvtepi8_epi16(_mm256_castsi256_si128(lhs)), _mm256_cvtepi8_epi16(_mm256_castsi256_si128(rhs))); - const __m256i highProducts = + const register_t lowProducts = _mm256_mullo_epi16(_mm256_cvtepi8_epi16(_mm256_castsi256_si128(lhs)), _mm256_cvtepi8_epi16(_mm256_castsi256_si128(rhs))); + const register_t highProducts = _mm256_mullo_epi16(_mm256_cvtepi8_epi16(_mm256_extracti128_si256(lhs, 1)), _mm256_cvtepi8_epi16(_mm256_extracti128_si256(rhs, 1))); - const __m256i interleavedSums = _mm256_hadd_epi16(lowProducts, highProducts); + const register_t interleavedSums = _mm256_hadd_epi16(lowProducts, highProducts); return _mm256_permute4x64_epi64(interleavedSums, _MM_SHUFFLE(3, 1, 2, 0)); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext256_mul_epi8(lhs, rhs); } /** @brief Divides corresponding signed 8-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epi8(lhs, rhs); } /** @brief Computes scalar-equivalent signed 8-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epi8(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { - auto sqrt16x16 = [](__m256i values) noexcept + auto sqrt16x16 = [](register_t values) noexcept { const __m128i low16 = _mm256_castsi256_si128(values); const __m128i high16 = _mm256_extracti128_si256(values, 1); - const __m256i low32 = _mm256_cvtepi16_epi32(low16); - const __m256i high32 = _mm256_cvtepi16_epi32(high16); - const __m256i lowRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_mm256_cvtepi32_ps(low32))); - const __m256i highRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_mm256_cvtepi32_ps(high32))); + const register_t low32 = _mm256_cvtepi16_epi32(low16); + const register_t high32 = _mm256_cvtepi16_epi32(high16); + const register_t lowRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_mm256_cvtepi32_ps(low32))); + const register_t highRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_mm256_cvtepi32_ps(high32))); const __m128i packedLow = _mm_packs_epi32(_mm256_castsi256_si128(lowRoots), _mm256_extracti128_si256(lowRoots, 1)); const __m128i packedHigh = _mm_packs_epi32(_mm256_castsi256_si128(highRoots), _mm256_extracti128_si256(highRoots, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(packedLow), packedHigh, 1); @@ -4044,16 +4129,16 @@ template <> struct SimdImpl256 const __m128i low8 = _mm256_castsi256_si128(lhs); const __m128i high8 = _mm256_extracti128_si256(lhs, 1); - const __m256i widenedLow = _mm256_cvtepi8_epi16(low8); - const __m256i widenedHigh = _mm256_cvtepi8_epi16(high8); - const __m256i rootsLow16 = sqrt16x16(widenedLow); - const __m256i rootsHigh16 = sqrt16x16(widenedHigh); + const register_t widenedLow = _mm256_cvtepi8_epi16(low8); + const register_t widenedHigh = _mm256_cvtepi8_epi16(high8); + const register_t rootsLow16 = sqrt16x16(widenedLow); + const register_t rootsHigh16 = sqrt16x16(widenedHigh); const __m128i packedLow = _mm_packs_epi16(_mm256_castsi256_si128(rootsLow16), _mm256_extracti128_si256(rootsLow16, 1)); const __m128i packedHigh = _mm_packs_epi16(_mm256_castsi256_si128(rootsHigh16), _mm256_extracti128_si256(rootsHigh16, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(packedLow), packedHigh, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -4061,7 +4146,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -4069,7 +4154,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -4084,88 +4169,88 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm256_abs_epi8(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi8(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_epi8(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_epi8(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _ext256_slli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _ext256_srli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext256_srai_epx8(lhs, rhs); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm256_adds_epi8(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm256_subs_epi8(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi8(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm256_set_epi8(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm256_setr_epi8(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm256_cmpgt_epi8(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtepi8_epi16(lhs, rhs); } @@ -4181,20 +4266,20 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 32)`. * @return Selected scalar lane. */ - static int8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static int8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 32, "Signed 8-bit extraction requires a valid 256-bit lane index"); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 2)); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 2)); const uint32_t selected_dword = static_cast(_mm_cvtsi128_si32(_mm256_castsi256_si128(selected))); return static_cast(selected_dword >> ((index & 3) * 8)); } /** @brief Replaces the compile-time-selected signed 8-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int8_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int8_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 8-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int8_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int8_t rhs) noexcept { return _mm256_insert_epi8(lhs, static_cast(rhs), index); } @@ -4205,37 +4290,35 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 32)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const int8_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int8_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 32, "Signed 8-bit insertion requires a valid 256-bit lane index"); - if (index < 16) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 16); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = + _mm256_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi8(lane_indices, _mm256_set1_epi8(static_cast(index))); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi8(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi8(lhs, rhs); } // misc /** @brief Shuffles bytes through the native runtime selector-register instruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept - requires(std::same_as && std::same_as) + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept + requires(std::same_as && std::same_as) { return _mm256_shuffle_epi8(lhs, rhs); } /** @brief Selects bytes through the native runtime mask-register operation. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const __m256i lhs, const __m256i rhs, const __m256i mask) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const register_t lhs, const register_t rhs, const register_t mask) noexcept { return register_blend_bytes(lhs, rhs, mask); } @@ -4247,8 +4330,10 @@ template <> struct SimdImpl256 template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects bytes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -4261,7 +4346,7 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 32 && ((indices < 32) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { constexpr auto selectors = std::array{indices...}; if constexpr (!logical_shuffle_256_has_cross_half_selector<16, selectors>()) @@ -4270,7 +4355,7 @@ template <> struct SimdImpl256 } else { - const __m256i swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); + const register_t swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); if constexpr (!logical_shuffle_256_has_local_half_selector<16, selectors>()) return _mm256_shuffle_epi8(swapped, make_logical_shuffle_256_byte_control<1, true, selectors>(std::make_index_sequence<32>{})); else @@ -4280,53 +4365,53 @@ template <> struct SimdImpl256 } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi8(lhs, rhs); } /** @brief Multiplies unsigned byte lanes and adds adjacent products into unsigned 16-bit lanes. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m256i lowProducts = _mm256_mullo_epi16(_mm256_cvtepu8_epi16(_mm256_castsi256_si128(lhs)), _mm256_cvtepu8_epi16(_mm256_castsi256_si128(rhs))); - const __m256i highProducts = + const register_t lowProducts = _mm256_mullo_epi16(_mm256_cvtepu8_epi16(_mm256_castsi256_si128(lhs)), _mm256_cvtepu8_epi16(_mm256_castsi256_si128(rhs))); + const register_t highProducts = _mm256_mullo_epi16(_mm256_cvtepu8_epi16(_mm256_extracti128_si256(lhs, 1)), _mm256_cvtepu8_epi16(_mm256_extracti128_si256(rhs, 1))); - const __m256i interleavedSums = _mm256_hadd_epi16(lowProducts, highProducts); + const register_t interleavedSums = _mm256_hadd_epi16(lowProducts, highProducts); return _mm256_permute4x64_epi64(interleavedSums, _MM_SHUFFLE(3, 1, 2, 0)); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext256_mul_epi8(lhs, rhs); } /** @brief Divides corresponding unsigned 8-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epu8(lhs, rhs); } /** @brief Computes scalar-equivalent unsigned 8-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epu8(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { - auto sqrt16x16 = [](__m256i values) noexcept + auto sqrt16x16 = [](register_t values) noexcept { const __m128i low16 = _mm256_castsi256_si128(values); const __m128i high16 = _mm256_extracti128_si256(values, 1); - const __m256i low32 = _mm256_cvtepu16_epi32(low16); - const __m256i high32 = _mm256_cvtepu16_epi32(high16); - const __m256i lowRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_ext256_cvtepu32_ps(low32))); - const __m256i highRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_ext256_cvtepu32_ps(high32))); + const register_t low32 = _mm256_cvtepu16_epi32(low16); + const register_t high32 = _mm256_cvtepu16_epi32(high16); + const register_t lowRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_ext256_cvtepu32_ps(low32))); + const register_t highRoots = _mm256_cvtps_epi32(_mm256_sqrt_ps(_ext256_cvtepu32_ps(high32))); const __m128i packedLow = _mm_packus_epi32(_mm256_castsi256_si128(lowRoots), _mm256_extracti128_si256(lowRoots, 1)); const __m128i packedHigh = _mm_packus_epi32(_mm256_castsi256_si128(highRoots), _mm256_extracti128_si256(highRoots, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(packedLow), packedHigh, 1); @@ -4334,16 +4419,16 @@ template <> struct SimdImpl256 const __m128i low8 = _mm256_castsi256_si128(lhs); const __m128i high8 = _mm256_extracti128_si256(lhs, 1); - const __m256i widenedLow = _mm256_cvtepu8_epi16(low8); - const __m256i widenedHigh = _mm256_cvtepu8_epi16(high8); - const __m256i rootsLow16 = sqrt16x16(widenedLow); - const __m256i rootsHigh16 = sqrt16x16(widenedHigh); + const register_t widenedLow = _mm256_cvtepu8_epi16(low8); + const register_t widenedHigh = _mm256_cvtepu8_epi16(high8); + const register_t rootsLow16 = sqrt16x16(widenedLow); + const register_t rootsHigh16 = sqrt16x16(widenedHigh); const __m128i packedLow = _mm_packus_epi16(_mm256_castsi256_si128(rootsLow16), _mm256_extracti128_si256(rootsLow16, 1)); const __m128i packedHigh = _mm_packus_epi16(_mm256_castsi256_si128(rootsHigh16), _mm256_extracti128_si256(rootsHigh16, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(packedLow), packedHigh, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -4351,7 +4436,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -4359,7 +4444,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -4374,93 +4459,93 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm256_abs_epi8(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi8(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_epu8(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_epu8(lhs, rhs); } /** @brief Computes lane-wise averages for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept { return _mm256_avg_epu8(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _ext256_slli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _ext256_srli_epx8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext256_srai_epx8(lhs, rhs); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm256_adds_epu8(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm256_subs_epu8(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _ext256_set1_epu8(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm256_set_epi8(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm256_setr_epi8(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext256_cmpgt_epu8(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtepu8_epi16(lhs, rhs); } @@ -4476,20 +4561,20 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 32)`. * @return Selected scalar lane. */ - static uint8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static uint8_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 32, "Unsigned 8-bit extraction requires a valid 256-bit lane index"); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 2)); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 2)); const uint32_t selected_dword = static_cast(_mm_cvtsi128_si32(_mm256_castsi256_si128(selected))); return static_cast(selected_dword >> ((index & 3) * 8)); } /** @brief Replaces the compile-time-selected unsigned 8-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint8_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint8_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 8-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint8_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint8_t rhs) noexcept { return _mm256_insert_epi8(lhs, static_cast(rhs), index); } @@ -4500,37 +4585,35 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 32)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const uint8_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint8_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 32, "Unsigned 8-bit insertion requires a valid 256-bit lane index"); - if (index < 16) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 16); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = + _mm256_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi8(lane_indices, _mm256_set1_epi8(static_cast(index))); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi8(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi8(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi8(lhs, rhs); } // misc /** @brief Shuffles bytes through the native runtime selector-register instruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept - requires(std::same_as && std::same_as) + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle(auto lhs, auto rhs) noexcept + requires(std::same_as && std::same_as) { return _mm256_shuffle_epi8(lhs, rhs); } /** @brief Selects bytes through the native runtime mask-register operation. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const __m256i lhs, const __m256i rhs, const __m256i mask) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) blend(const register_t lhs, const register_t rhs, const register_t mask) noexcept { return register_blend_bytes(lhs, rhs, mask); } @@ -4542,8 +4625,10 @@ template <> struct SimdImpl256 template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects 16-bit lanes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -4556,7 +4641,7 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 16 && ((indices < 16) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { constexpr auto selectors = std::array{indices...}; if constexpr (!logical_shuffle_256_has_cross_half_selector<8, selectors>()) @@ -4565,7 +4650,7 @@ template <> struct SimdImpl256 } else { - const __m256i swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); + const register_t swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); if constexpr (!logical_shuffle_256_has_local_half_selector<8, selectors>()) return _mm256_shuffle_epi8(swapped, make_logical_shuffle_256_byte_control<2, true, selectors>(std::make_index_sequence<32>{})); else @@ -4575,46 +4660,46 @@ template <> struct SimdImpl256 } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi16(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { return _mm256_madd_epi16(lhs, rhs); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm256_mullo_epi16(lhs, rhs); } /** @brief Divides corresponding signed 16-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epi16(lhs, rhs); } /** @brief Computes scalar-equivalent signed 16-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epi16(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { auto sqrt16x8 = [](__m128i values) noexcept { - const __m256i widened = _mm256_cvtepi16_epi32(values); + const register_t widened = _mm256_cvtepi16_epi32(values); const __m256 roots = _mm256_sqrt_ps(_mm256_cvtepi32_ps(widened)); - const __m256i integers = _mm256_cvtps_epi32(roots); + const register_t integers = _mm256_cvtps_epi32(roots); return _mm_packs_epi32(_mm256_castsi256_si128(integers), _mm256_extracti128_si256(integers, 1)); }; @@ -4625,7 +4710,7 @@ template <> struct SimdImpl256 return _mm256_inserti128_si256(_mm256_castsi128_si256(rootsLow), rootsHigh, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -4633,7 +4718,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -4641,7 +4726,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -4656,92 +4741,92 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm256_abs_epi16(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi16(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_epi16(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_epi16(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm256_slli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm256_srli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm256_srai_epi16(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hadd_epi16(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hsub_epi16(lhs, rhs); } /** @brief Horizontally adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept { return _mm256_hadds_epi16(lhs, rhs); } /** @brief Horizontally subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept { return _mm256_hsubs_epi16(lhs, rhs); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm256_adds_epi16(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm256_subs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept { const __m128i lhsLo128 = _mm256_castsi256_si128(lhs); const __m128i rhsLo128 = _mm256_castsi256_si128(rhs); const __m128i lhsHi128 = _mm256_extracti128_si256(lhs, 1); const __m128i rhsHi128 = _mm256_extracti128_si256(rhs, 1); - const __m256i loProducts = _mm256_mullo_epi32(_mm256_cvtepi16_epi32(lhsLo128), _mm256_cvtepi16_epi32(rhsLo128)); - const __m256i hiProducts = _mm256_mullo_epi32(_mm256_cvtepi16_epi32(lhsHi128), _mm256_cvtepi16_epi32(rhsHi128)); + const register_t loProducts = _mm256_mullo_epi32(_mm256_cvtepi16_epi32(lhsLo128), _mm256_cvtepi16_epi32(rhsLo128)); + const register_t hiProducts = _mm256_mullo_epi32(_mm256_cvtepi16_epi32(lhsHi128), _mm256_cvtepi16_epi32(rhsHi128)); const __m128i loPacked = _mm_packs_epi32(_mm256_castsi256_si128(loProducts), _mm256_extracti128_si256(loProducts, 1)); const __m128i hiPacked = _mm_packs_epi32(_mm256_castsi256_si128(hiProducts), _mm256_extracti128_si256(hiProducts, 1)); @@ -4749,35 +4834,35 @@ template <> struct SimdImpl256 } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi16(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args &&...args) noexcept { return _mm256_set_epi16(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args &&...args) noexcept { return _mm256_setr_epi16(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm256_cmpgt_epi16(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtepi16_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm256_packs_epi16(lhs, rhs); } @@ -4793,20 +4878,20 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 16)`. * @return Selected scalar lane. */ - static int16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static int16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Signed 16-bit extraction requires a valid 256-bit lane index"); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 1)); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 1)); const uint32_t selected_dword = static_cast(_mm_cvtsi128_si32(_mm256_castsi256_si128(selected))); return static_cast(selected_dword >> ((index & 1) * 16)); } /** @brief Replaces the compile-time-selected signed 16-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int16_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int16_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 16-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int16_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int16_t rhs) noexcept { return _mm256_insert_epi16(lhs, static_cast(rhs), index); } @@ -4817,24 +4902,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 16)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const int16_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int16_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Signed 16-bit insertion requires a valid 256-bit lane index"); - if (index < 8) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 8); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = _mm256_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi16(lane_indices, _mm256_set1_epi16(static_cast(index))); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi16(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi16(lhs, rhs); } @@ -4845,12 +4927,12 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), false); } /** @brief Shuffles the low four signed 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(__m256i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(register_t lhs) noexcept { return _mm256_shufflelo_epi16(lhs, imm8); } @@ -4859,12 +4941,12 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), true); } /** @brief Shuffles the high four signed 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(__m256i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(register_t lhs) noexcept { return _mm256_shufflehi_epi16(lhs, imm8); } @@ -4874,23 +4956,30 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << (index % 8)); + }); + const register_t selected = _mm256_and_si256(_mm256_set1_epi16(static_cast(imm8)), selectors); + return _mm256_blendv_epi8(lhs, rhs, _mm256_cmpeq_epi16(selected, selectors)); } /** @brief Selects signed 16-bit lanes from two 256-bit registers with a repeated immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m256i lhs, const __m256i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm256_blend_epi16(lhs, rhs, imm8); } }; template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects 16-bit lanes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -4903,7 +4992,7 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 16 && ((indices < 16) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { constexpr auto selectors = std::array{indices...}; if constexpr (!logical_shuffle_256_has_cross_half_selector<8, selectors>()) @@ -4912,7 +5001,7 @@ template <> struct SimdImpl256 } else { - const __m256i swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); + const register_t swapped = _mm256_permute2x128_si256(lhs, lhs, 0x01); if constexpr (!logical_shuffle_256_has_local_half_selector<8, selectors>()) return _mm256_shuffle_epi8(swapped, make_logical_shuffle_256_byte_control<2, true, selectors>(std::make_index_sequence<32>{})); else @@ -4922,50 +5011,50 @@ template <> struct SimdImpl256 } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi16(lhs, rhs); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } /** @brief Multiplies adjacent unsigned 16-bit lanes and adds their products into unsigned 32-bit lanes. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m256i lowProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(_mm256_castsi256_si128(lhs)), _mm256_cvtepu16_epi32(_mm256_castsi256_si128(rhs))); - const __m256i highProducts = + const register_t lowProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(_mm256_castsi256_si128(lhs)), _mm256_cvtepu16_epi32(_mm256_castsi256_si128(rhs))); + const register_t highProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(_mm256_extracti128_si256(lhs, 1)), _mm256_cvtepu16_epi32(_mm256_extracti128_si256(rhs, 1))); - const __m256i interleavedSums = _mm256_hadd_epi32(lowProducts, highProducts); + const register_t interleavedSums = _mm256_hadd_epi32(lowProducts, highProducts); return _mm256_permute4x64_epi64(interleavedSums, _MM_SHUFFLE(3, 1, 2, 0)); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm256_mullo_epi16(lhs, rhs); } /** @brief Divides corresponding unsigned 16-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epu16(lhs, rhs); } /** @brief Computes scalar-equivalent unsigned 16-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epu16(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { auto sqrt16x8 = [](__m128i values) noexcept { - const __m256i widened = _mm256_cvtepu16_epi32(values); + const register_t widened = _mm256_cvtepu16_epi32(values); const __m256 roots = _mm256_sqrt_ps(_ext256_cvtepu32_ps(widened)); - const __m256i integers = _mm256_cvtps_epi32(roots); + const register_t integers = _mm256_cvtps_epi32(roots); return _mm_packus_epi32(_mm256_castsi256_si128(integers), _mm256_extracti128_si256(integers, 1)); }; @@ -4976,7 +5065,7 @@ template <> struct SimdImpl256 return _mm256_inserti128_si256(_mm256_castsi128_si256(rootsLow), rootsHigh, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -4984,7 +5073,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -4992,7 +5081,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -5007,103 +5096,103 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm256_abs_epi16(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi16(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_epu16(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_epu16(lhs, rhs); } /** @brief Computes lane-wise averages for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) avg(auto lhs, auto rhs) noexcept { return _mm256_avg_epu16(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm256_slli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm256_srli_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm256_srai_epi16(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hadd_epi16(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hsub_epi16(lhs, rhs); } /** @brief Horizontally adds unsigned 16-bit lanes with unsigned saturation in each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hadd_saturated(auto lhs, auto rhs) noexcept { - const __m256i zero = _mm256_setzero_si256(); - const __m256i lhsPairs = _mm256_adds_epu16(lhs, _mm256_srli_epi32(lhs, 16)); - const __m256i rhsPairs = _mm256_adds_epu16(rhs, _mm256_srli_epi32(rhs, 16)); + const register_t zero = _mm256_setzero_si256(); + const register_t lhsPairs = _mm256_adds_epu16(lhs, _mm256_srli_epi32(lhs, 16)); + const register_t rhsPairs = _mm256_adds_epu16(rhs, _mm256_srli_epi32(rhs, 16)); return _mm256_packus_epi32(_mm256_blend_epi16(lhsPairs, zero, 0xAA), _mm256_blend_epi16(rhsPairs, zero, 0xAA)); } /** @brief Horizontally subtracts unsigned 16-bit lanes with unsigned saturation in each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) hsubtract_saturated(auto lhs, auto rhs) noexcept { - const __m256i zero = _mm256_setzero_si256(); - const __m256i lhsPairs = _mm256_subs_epu16(lhs, _mm256_srli_epi32(lhs, 16)); - const __m256i rhsPairs = _mm256_subs_epu16(rhs, _mm256_srli_epi32(rhs, 16)); + const register_t zero = _mm256_setzero_si256(); + const register_t lhsPairs = _mm256_subs_epu16(lhs, _mm256_srli_epi32(lhs, 16)); + const register_t rhsPairs = _mm256_subs_epu16(rhs, _mm256_srli_epi32(rhs, 16)); return _mm256_packus_epi32(_mm256_blend_epi16(lhsPairs, zero, 0xAA), _mm256_blend_epi16(rhsPairs, zero, 0xAA)); } // arithmetic (saturated) /** @brief Adds lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_saturated(auto lhs, auto rhs) noexcept { return _mm256_adds_epu16(lhs, rhs); } /** @brief Subtracts lanes with saturation for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_saturated(auto lhs, auto rhs) noexcept { return _mm256_subs_epu16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) multiply_saturated(auto lhs, auto rhs) noexcept { const __m128i lhsLo128 = _mm256_castsi256_si128(lhs); const __m128i rhsLo128 = _mm256_castsi256_si128(rhs); const __m128i lhsHi128 = _mm256_extracti128_si256(lhs, 1); const __m128i rhsHi128 = _mm256_extracti128_si256(rhs, 1); - const __m256i loProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(lhsLo128), _mm256_cvtepu16_epi32(rhsLo128)); - const __m256i hiProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(lhsHi128), _mm256_cvtepu16_epi32(rhsHi128)); + const register_t loProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(lhsLo128), _mm256_cvtepu16_epi32(rhsLo128)); + const register_t hiProducts = _mm256_mullo_epi32(_mm256_cvtepu16_epi32(lhsHi128), _mm256_cvtepu16_epi32(rhsHi128)); const __m128i loPacked = _mm_packus_epi32(_mm256_castsi256_si128(loProducts), _mm256_extracti128_si256(loProducts, 1)); const __m128i hiPacked = _mm_packus_epi32(_mm256_castsi256_si128(hiProducts), _mm256_extracti128_si256(hiProducts, 1)); @@ -5111,35 +5200,35 @@ template <> struct SimdImpl256 } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi16(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_epi16(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_epi16(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext256_cmpgt_epu16(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtepu16_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm256_packus_epi16(lhs, rhs); } @@ -5155,20 +5244,20 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 16)`. * @return Selected scalar lane. */ - static uint16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static uint16_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Unsigned 16-bit extraction requires a valid 256-bit lane index"); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 1)); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index >> 1)); const uint32_t selected_dword = static_cast(_mm_cvtsi128_si32(_mm256_castsi256_si128(selected))); return static_cast(selected_dword >> ((index & 1) * 16)); } /** @brief Replaces the compile-time-selected unsigned 16-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint16_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint16_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 16-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint16_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint16_t rhs) noexcept { return _mm256_insert_epi16(lhs, static_cast(rhs), index); } @@ -5179,24 +5268,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 16)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const uint16_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint16_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 16, "Unsigned 16-bit insertion requires a valid 256-bit lane index"); - if (index < 8) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 8); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = _mm256_setr_epi16(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi16(lane_indices, _mm256_set1_epi16(static_cast(index))); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi16(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi16(lhs, rhs); } @@ -5207,12 +5293,12 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), false); } /** @brief Shuffles the low four unsigned 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(__m256i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_lo(register_t lhs) noexcept { return _mm256_shufflelo_epi16(lhs, imm8); } @@ -5221,12 +5307,12 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_half_16_slow(lhs, static_cast(rhs), true); } /** @brief Shuffles the high four unsigned 16-bit lanes in each 128-bit group with an immediate control. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(__m256i lhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle_hi(register_t lhs) noexcept { return _mm256_shufflehi_epi16(lhs, imm8); } @@ -5236,23 +5322,30 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << (index % 8)); + }); + const register_t selected = _mm256_and_si256(_mm256_set1_epi16(static_cast(imm8)), selectors); + return _mm256_blendv_epi8(lhs, rhs, _mm256_cmpeq_epi16(selected, selectors)); } /** @brief Selects unsigned 16-bit lanes from two 256-bit registers with a repeated immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m256i lhs, const __m256i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm256_blend_epi16(lhs, rhs, imm8); } }; template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects 32-bit lanes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -5265,54 +5358,54 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 8 && ((indices < 8) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm256_permutevar8x32_epi32(lhs, _mm256_setr_epi32(static_cast(indices)...)); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi32(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m256i evenProducts = _mm256_mul_epi32(lhs, rhs); - const __m256i oddProducts = _mm256_mul_epi32(_mm256_srli_si256(lhs, 4), _mm256_srli_si256(rhs, 4)); + const register_t evenProducts = _mm256_mul_epi32(lhs, rhs); + const register_t oddProducts = _mm256_mul_epi32(_mm256_srli_si256(lhs, 4), _mm256_srli_si256(rhs, 4)); return _mm256_add_epi64(evenProducts, oddProducts); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm256_mullo_epi32(lhs, rhs); } /** @brief Divides corresponding signed 32-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epi32(lhs, rhs); } /** @brief Computes scalar-equivalent signed 32-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epi32(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m256 roots = _mm256_sqrt_ps(_mm256_cvtepi32_ps(lhs)); return _mm256_cvtps_epi32(roots); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -5320,7 +5413,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -5328,7 +5421,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -5343,92 +5436,92 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm256_abs_epi32(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi32(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_epi32(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_epi32(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm256_slli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm256_srli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm256_srai_epi32(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hadd_epi32(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hsub_epi32(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi32(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_epi32(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_epi32(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm256_cmpgt_epi32(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtepi32_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm256_packs_epi32(lhs, rhs); } @@ -5444,19 +5537,19 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 8)`. * @return Selected scalar lane. */ - static int32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static int32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Signed 32-bit extraction requires a valid 256-bit lane index"); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index)); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index)); return _mm_cvtsi128_si32(_mm256_castsi256_si128(selected)); } /** @brief Replaces the compile-time-selected signed 32-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int32_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int32_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 32-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int32_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int32_t rhs) noexcept { return _mm256_insert_epi32(lhs, rhs, index); } @@ -5467,24 +5560,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 8)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const int32_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int32_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Signed 32-bit insertion requires a valid 256-bit lane index"); - if (index < 4) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 4); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = _mm256_setr_epi32(0, 1, 2, 3, 4, 5, 6, 7); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi32(lane_indices, _mm256_set1_epi32(index)); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi32(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi32(lhs, rhs); } @@ -5495,7 +5585,7 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_32_slow(lhs, static_cast(rhs)); } @@ -5504,7 +5594,7 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_32_slow(lhs, static_cast(rhs)); } @@ -5514,23 +5604,30 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const register_t selected = _mm256_and_si256(_mm256_set1_epi32(imm8), selectors); + return _mm256_blendv_epi8(lhs, rhs, _mm256_cmpeq_epi32(selected, selectors)); } /** @brief Selects signed 32-bit lanes from two 256-bit registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m256i lhs, const __m256i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm256_blend_epi32(lhs, rhs, imm8); } }; template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects 32-bit lanes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -5543,13 +5640,13 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 8 && ((indices < 8) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm256_permutevar8x32_epi32(lhs, _mm256_setr_epi32(static_cast(indices)...)); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi32(lhs, rhs); } @@ -5559,42 +5656,42 @@ template <> struct SimdImpl256 * @param lhs The unsigned integer lanes. * @return The converted floating-point lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) convert_to_float(const __m256i lhs) noexcept + static auto SIMD_FLAGS(InOut, ForceInline) convert_to_float(const register_t lhs) noexcept { return _ext256_cvtepu32_ps(lhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { - const __m256i evenProducts = _mm256_mul_epu32(lhs, rhs); - const __m256i oddProducts = _mm256_mul_epu32(_mm256_srli_si256(lhs, 4), _mm256_srli_si256(rhs, 4)); + const register_t evenProducts = _mm256_mul_epu32(lhs, rhs); + const register_t oddProducts = _mm256_mul_epu32(_mm256_srli_si256(lhs, 4), _mm256_srli_si256(rhs, 4)); return _mm256_add_epi64(evenProducts, oddProducts); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm256_mullo_epi32(lhs, rhs); } /** @brief Divides corresponding unsigned 32-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epu32(lhs, rhs); } /** @brief Computes scalar-equivalent unsigned 32-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epu32(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128i low = _mm256_castsi256_si128(lhs); const __m128i high = _mm256_extracti128_si256(lhs, 1); @@ -5605,7 +5702,7 @@ template <> struct SimdImpl256 return _mm256_inserti128_si256(_mm256_castsi128_si256(lowInts), highInts, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -5613,7 +5710,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -5621,7 +5718,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -5636,92 +5733,92 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _mm256_abs_epi32(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi32(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_epu32(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_epu32(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm256_slli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm256_srli_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _mm256_srai_epi32(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hadd_epi32(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hsub_epi32(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi32(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_epi32(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_epi32(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext256_cmpgt_epu32(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtepu32_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) compress(auto lhs, auto rhs) noexcept { return _mm256_packus_epi32(lhs, rhs); } @@ -5737,19 +5834,19 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 8)`. * @return Selected scalar lane. */ - static uint32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static uint32_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Unsigned 32-bit extraction requires a valid 256-bit lane index"); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index)); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_set1_epi32(index)); return static_cast(_mm_cvtsi128_si32(_mm256_castsi256_si128(selected))); } /** @brief Replaces the compile-time-selected unsigned 32-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint32_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint32_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 32-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint32_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint32_t rhs) noexcept { return _mm256_insert_epi32(lhs, std::bit_cast(rhs), index); } @@ -5760,24 +5857,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 8)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const uint32_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint32_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "Unsigned 32-bit insertion requires a valid 256-bit lane index"); - if (index < 4) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 4); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = _mm256_setr_epi32(0, 1, 2, 3, 4, 5, 6, 7); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi32(lane_indices, _mm256_set1_epi32(index)); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi32(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi32(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi32(lhs, rhs); } @@ -5788,7 +5882,7 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each low four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_lo_slow(auto lhs, auto rhs) noexcept { return register_shuffle_32_slow(lhs, static_cast(rhs)); } @@ -5797,7 +5891,7 @@ template <> struct SimdImpl256 * @param rhs Runtime control byte. * @return Register with each high four-lane group shuffled. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) shuffle_hi_slow(auto lhs, auto rhs) noexcept { return register_shuffle_32_slow(lhs, static_cast(rhs)); } @@ -5807,23 +5901,30 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const register_t selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const register_t selected = _mm256_and_si256(_mm256_set1_epi32(imm8), selectors); + return _mm256_blendv_epi8(lhs, rhs, _mm256_cmpeq_epi32(selected, selectors)); } /** @brief Selects unsigned 32-bit lanes from two 256-bit registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m256i lhs, const __m256i rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm256_blend_epi32(lhs, rhs, imm8); } }; template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects 64-bit lanes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -5836,55 +5937,55 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 4 && ((indices < 4) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm256_permute4x64_epi64(lhs, encode_logical_shuffle_32_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi64(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { const __m128i low = SimdImpl128::multiply_add_adjacent(_mm256_castsi256_si128(lhs), _mm256_castsi256_si128(rhs)); const __m128i high = SimdImpl128::multiply_add_adjacent(_mm256_extracti128_si256(lhs, 1), _mm256_extracti128_si256(rhs, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(low), high, 1); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext256_mullo_epi64(lhs, rhs); } /** @brief Divides corresponding signed 64-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epi64(lhs, rhs); } /** @brief Computes scalar-equivalent signed 64-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epi64(lhs, rhs); } /** @brief Computes integer square roots lane-wise using register extracts and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128i lowRoots = SimdImpl128::sqrt(_mm256_castsi256_si128(lhs)); const __m128i highRoots = SimdImpl128::sqrt(_mm256_extracti128_si256(lhs, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(lowRoots), highRoots, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -5892,7 +5993,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -5900,7 +6001,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -5915,76 +6016,76 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _ext256_abs_epi64(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi64(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _ext256_min_epi64(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _ext256_max_epi64(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm256_slli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm256_srli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext256_srai_epi64(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi64x(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_epi64x(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_epi64x(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _mm256_cmpgt_epi64(lhs, rhs); } // conversion - // static auto SIMD_FLAGS(InOut, ForceInline) expand (auto lhs, auto rhs) noexcept { return _mm256_cvtepi64_epi128(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) expand (auto lhs, auto rhs) noexcept { return _mm256_cvtepi64_epi128(lhs, rhs); } // extract / insert template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline) extract(auto lhs) noexcept @@ -5997,21 +6098,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 4)`. * @return Selected scalar lane. */ - static int64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static int64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Signed 64-bit extraction requires a valid 256-bit lane index"); - const __m256i first_word = _mm256_set1_epi32(index * 2); - const __m256i word_offsets = _mm256_setr_epi32(0, 1, 0, 1, 0, 1, 0, 1); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_add_epi32(first_word, word_offsets)); + const register_t first_word = _mm256_set1_epi32(index * 2); + const register_t word_offsets = _mm256_setr_epi32(0, 1, 0, 1, 0, 1, 0, 1); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_add_epi32(first_word, word_offsets)); return SimdImpl128::template extract<0>(_mm256_castsi256_si128(selected)); } /** @brief Replaces the compile-time-selected signed 64-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const int64_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const int64_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected signed 64-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int64_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const int64_t rhs) noexcept { return _mm256_insert_epi64(lhs, rhs, index); } @@ -6022,24 +6123,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 4)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const int64_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const int64_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Signed 64-bit insertion requires a valid 256-bit lane index"); - if (index < 2) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 2); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = _mm256_setr_epi64x(0, 1, 2, 3); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi64(lane_indices, _mm256_set1_epi64x(index)); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi64x(rhs), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi64(lhs, rhs); } @@ -6047,8 +6145,10 @@ template <> struct SimdImpl256 template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256i; /** @brief Selects 64-bit lanes from two registers using a canonical predicate register. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256i condition, __m256i when_true, __m256i when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_epi8(when_false, when_true, condition); } @@ -6061,55 +6161,55 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 4 && ((indices < 4) && ...)) - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256i lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm256_permute4x64_epi64(lhs, encode_logical_shuffle_32_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_epi64(lhs, rhs); } /** @brief Multiplies adjacent lanes and adds their products for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_adjacent(auto lhs, auto rhs) noexcept { const __m128i low = SimdImpl128::multiply_add_adjacent(_mm256_castsi256_si128(lhs), _mm256_castsi256_si128(rhs)); const __m128i high = SimdImpl128::multiply_add_adjacent(_mm256_extracti128_si256(lhs, 1), _mm256_extracti128_si256(rhs, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(low), high, 1); } /** @brief Multiplies unsigned and signed byte pairs for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add_unsigned_signed_bytes(auto lhs, auto rhs) noexcept { return _mm256_maddubs_epi16(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _ext256_mullo_epi64(lhs, rhs); } /** @brief Divides corresponding unsigned 64-bit lanes with scalar instructions and intrinsic reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _ext256_div_epu64(lhs, rhs); } /** @brief Computes scalar-equivalent unsigned 64-bit remainders with register-only extraction and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) modulus(auto lhs, auto rhs) noexcept { return _ext256_rem_epu64(lhs, rhs); } /** @brief Computes integer square roots lane-wise using register extracts and reconstruction. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { const __m128i lowRoots = SimdImpl128::sqrt(_mm256_castsi256_si128(lhs)); const __m128i highRoots = SimdImpl128::sqrt(_mm256_extracti128_si256(lhs, 1)); return _mm256_inserti128_si256(_mm256_castsi128_si256(lowRoots), highRoots, 1); } /** @brief Computes one unchecked magnitude in lane zero of each 128-bit group. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude(_mm256_extracti128_si256(lhs, 1)); @@ -6117,7 +6217,7 @@ template <> struct SimdImpl256 } /** @brief Computes saturated magnitudes and adjacent overflow masks for both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude_checked(auto lhs) noexcept { const __m128i lowMagnitude = SimdImpl128::magnitude_checked(_mm256_castsi256_si128(lhs)); const __m128i highMagnitude = SimdImpl128::magnitude_checked(_mm256_extracti128_si256(lhs, 1)); @@ -6125,7 +6225,7 @@ template <> struct SimdImpl256 } /** @brief Returns the minimum value and its first lane position without materializing register data in memory. */ - static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept + static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) min_position(auto lhs) noexcept { const __m128i lowMeta = SimdImpl128::min_position(_mm256_castsi256_si128(lhs)); const __m128i highMeta = SimdImpl128::min_position(_mm256_extracti128_si256(lhs, 1)); @@ -6140,76 +6240,76 @@ template <> struct SimdImpl256 return _mm256_zextsi128_si256(output); } /** @brief Computes byte-wise absolute-difference sums for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sum_absolute_byte_differences(auto lhs, auto rhs) noexcept { return _mm256_sad_epu8(lhs, rhs); } /** @brief Computes immediate-selected byte-window absolute-difference sums for this native register specialization. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(__m256i lhs, __m256i rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multi_sum_absolute_byte_differences(register_t lhs, register_t rhs) noexcept { return _mm256_mpsadbw_epu8(lhs, rhs, imm8); } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return lhs; } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_epi64(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _ext256_min_epu64(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _ext256_max_epu64(lhs, rhs); } // shifting - static auto SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_left(auto lhs, auto rhs) noexcept { return _mm256_slli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right(auto lhs, auto rhs) noexcept { return _mm256_srli_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shift_right_arithmetic(auto lhs, auto rhs) noexcept { return _ext256_srai_epi64(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_epi64x(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_epi64x(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_epi64x(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _mm256_cmpeq_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext256_cmpgt_epu64(lhs, rhs); } // conversion - // static auto SIMD_FLAGS(InOut, ForceInline) expand (auto lhs, auto rhs) noexcept { return _mm256_cvtepu64_epi128(lhs, rhs); } + // static register_t SIMD_FLAGS(InOut, ForceInline) expand (auto lhs, auto rhs) noexcept { return _mm256_cvtepu64_epi128(lhs, rhs); } // extract / insert template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline) extract(auto lhs) noexcept @@ -6222,21 +6322,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 4)`. * @return Selected scalar lane. */ - static uint64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256i lhs, const int index) noexcept + static uint64_t SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Unsigned 64-bit extraction requires a valid 256-bit lane index"); - const __m256i first_word = _mm256_set1_epi32(index * 2); - const __m256i word_offsets = _mm256_setr_epi32(0, 1, 0, 1, 0, 1, 0, 1); - const __m256i selected = _mm256_permutevar8x32_epi32(lhs, _mm256_add_epi32(first_word, word_offsets)); + const register_t first_word = _mm256_set1_epi32(index * 2); + const register_t word_offsets = _mm256_setr_epi32(0, 1, 0, 1, 0, 1, 0, 1); + const register_t selected = _mm256_permutevar8x32_epi32(lhs, _mm256_add_epi32(first_word, word_offsets)); return SimdImpl128::template extract<0>(_mm256_castsi256_si128(selected)); } /** @brief Replaces the compile-time-selected unsigned 64-bit lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const uint64_t rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const uint64_t rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected unsigned 64-bit lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint64_t rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const uint64_t rhs) noexcept { return _mm256_insert_epi64(lhs, std::bit_cast(rhs), index); } @@ -6247,24 +6347,21 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 4)`. * @return Register with the selected lane replaced. */ - static __m256i SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256i lhs, const uint64_t rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const uint64_t rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "Unsigned 64-bit insertion requires a valid 256-bit lane index"); - if (index < 2) - { - const __m128i lower = SimdImpl128::insert_slow(_mm256_castsi256_si128(lhs), rhs, index); - return _mm256_inserti128_si256(lhs, lower, 0); - } - const __m128i upper = SimdImpl128::insert_slow(_mm256_extracti128_si256(lhs, 1), rhs, index - 2); - return _mm256_inserti128_si256(lhs, upper, 1); + const register_t lane_indices = _mm256_setr_epi64x(0, 1, 2, 3); + // A full-width equality mask selects the requested lane without branching on the 128-bit half. + const register_t selected_lane = _mm256_cmpeq_epi64(lane_indices, _mm256_set1_epi64x(index)); + return _mm256_blendv_epi8(lhs, _mm256_set1_epi64x(std::bit_cast(rhs)), selected_lane); } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_epi64(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_epi64(lhs, rhs); } @@ -6272,8 +6369,10 @@ template <> struct SimdImpl256 template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256; /** @brief Selects float lanes from two registers using a canonical predicate register. */ - static __m256 SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256 condition, __m256 when_true, __m256 when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_ps(when_false, when_true, condition); } @@ -6286,45 +6385,45 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 8 && ((indices < 8) && ...)) - static __m256 SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256 lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm256_permutevar8x32_ps(lhs, _mm256_setr_epi32(static_cast(indices)...)); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_ps(lhs, rhs); } /** @brief Alternates lane subtraction and addition for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept { return _mm256_addsub_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm256_mul_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _mm256_div_ps(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { return _mm256_sqrt_ps(lhs); } /** @brief Computes and broadcasts floating-point magnitudes independently in both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { return _mm256_sqrt_ps(_mm256_dp_ps(lhs, lhs, 0xFF)); } /** @brief Multiplies lanes and adds a third register for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept { #if SIMDLIB_HAS_FMA return _mm256_fmadd_ps(lhs, rhs, addend); @@ -6333,7 +6432,7 @@ template <> struct SimdImpl256 #endif } /** @brief Computes an immediate-controlled dot product for this native register specialization. */ - template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(__m256 lhs, __m256 rhs) noexcept + template static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(register_t lhs, register_t rhs) noexcept { const __m128 lhsLow = _mm256_castps256_ps128(lhs); const __m128 lhsHigh = _mm256_extractf128_ps(lhs, 1); @@ -6345,63 +6444,63 @@ template <> struct SimdImpl256 } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _ext256_abs_ps(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_ps(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_ps(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_ps(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hadd_ps(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hsub_ps(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_ps(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_ps(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_ps(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _ext256_cmpeq_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext256_cmpgt_ps(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtps_epi32(lhs, rhs); } @@ -6427,19 +6526,19 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 8)`. * @return Selected scalar lane. */ - static float SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256 lhs, const int index) noexcept + static float SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "32-bit floating-point extraction requires a valid 256-bit lane index"); - const __m256 selected = _mm256_permutevar8x32_ps(lhs, _mm256_set1_epi32(index)); + const register_t selected = _mm256_permutevar8x32_ps(lhs, _mm256_set1_epi32(index)); return SimdImpl128::template extract<0>(_mm256_castps256_ps128(selected)); } /** @brief Replaces the compile-time-selected 32-bit floating-point lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const float rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const float rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected 32-bit floating-point lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const float rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const float rhs) noexcept { constexpr int half_index = index / 4; constexpr int lane_index = index % 4; @@ -6458,7 +6557,7 @@ template <> struct SimdImpl256 * @param index Selected lane index. * @return Register with the selected lane replaced. */ - static __m256 SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256 lhs, const float rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const float rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 8, "32-bit floating-point insertion requires a valid 256-bit lane index"); if (index < 4) @@ -6471,11 +6570,11 @@ template <> struct SimdImpl256 } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_ps(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_ps(lhs, rhs); } @@ -6487,7 +6586,7 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the shuffled lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, const int imm8) noexcept { return register_shuffle_float_slow(lhs, rhs, imm8); } @@ -6497,23 +6596,30 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const __m256i selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const __m256i selected = _mm256_and_si256(_mm256_set1_epi32(imm8), selectors); + return _mm256_blendv_ps(lhs, rhs, _mm256_castsi256_ps(_mm256_cmpeq_epi32(selected, selectors))); } /** @brief Selects 32-bit floating-point lanes from two 256-bit registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m256 lhs, const __m256 rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm256_blend_ps(lhs, rhs, imm8); } }; template <> struct SimdImpl256 { + /** @brief Native SIMD register returned by this implementation backend. */ + using register_t = __m256d; /** @brief Selects double lanes from two registers using a canonical predicate register. */ - static __m256d SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(__m256d condition, __m256d when_true, __m256d when_false) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) select(register_t condition, register_t when_true, register_t when_false) noexcept { return _mm256_blendv_pd(when_false, when_true, condition); } @@ -6526,46 +6632,46 @@ template <> struct SimdImpl256 */ template requires(sizeof...(indices) == 4 && ((indices < 4) && ...)) - static __m256d SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(__m256d lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) shuffle(register_t lhs) noexcept { return _mm256_permute4x64_pd(lhs, encode_logical_shuffle_32_immediate()); } // arithmetic - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add(auto lhs, auto rhs) noexcept { return _mm256_add_pd(lhs, rhs); } /** @brief Alternates lane subtraction and addition for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_subtract(auto lhs, auto rhs) noexcept { return _mm256_addsub_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract(auto lhs, auto rhs) noexcept { return _mm256_sub_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply(auto lhs, auto rhs) noexcept { return _mm256_mul_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) divide(auto lhs, auto rhs) noexcept { return _mm256_div_pd(lhs, rhs); } /** @brief Computes lane-wise square roots for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) sqrt(auto lhs) noexcept { return _mm256_sqrt_pd(lhs); } /** @brief Computes and broadcasts floating-point magnitudes independently in both 128-bit groups. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) magnitude(auto lhs) noexcept { - const __m256d squares = _mm256_mul_pd(lhs, lhs); + const register_t squares = _mm256_mul_pd(lhs, lhs); return _mm256_sqrt_pd(_mm256_hadd_pd(squares, squares)); } /** @brief Multiplies lanes and adds a third register for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) multiply_add(auto lhs, auto rhs, auto addend) noexcept { #if SIMDLIB_HAS_FMA return _mm256_fmadd_pd(lhs, rhs, addend); @@ -6574,7 +6680,7 @@ template <> struct SimdImpl256 #endif } /** @brief Computes an immediate-controlled dot product for this native register specialization. */ - template static auto SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(__m256d lhs, __m256d rhs) noexcept + template static register_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) dot_product(register_t lhs, register_t rhs) noexcept { const __m128d lhsLow = _mm256_castpd256_pd128(lhs); const __m128d lhsHigh = _mm256_extractf128_pd(lhs, 1); @@ -6586,63 +6692,63 @@ template <> struct SimdImpl256 } // /** @brief Computes lane-wise absolute values for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) absolute(auto lhs) noexcept { return _ext256_abs_pd(lhs); } - static auto SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) negate(auto lhs, auto rhs) noexcept { return _mm256_sub_pd(lhs, rhs); } /** @brief Computes lane-wise minima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) min(auto lhs, auto rhs) noexcept { return _mm256_min_pd(lhs, rhs); } /** @brief Computes lane-wise maxima for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) max(auto lhs, auto rhs) noexcept { return _mm256_max_pd(lhs, rhs); } // arithmetic (horizontal) /** @brief Horizontally adds adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) add_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hadd_pd(lhs, rhs); } /** @brief Horizontally subtracts adjacent lanes for this native register specialization. */ - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) subtract_horizontal(auto lhs, auto rhs) noexcept { return _mm256_hsub_pd(lhs, rhs); } // loading - static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept + static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set1(auto lhs) noexcept { return _mm256_set1_pd(lhs); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) set(Args... args) noexcept { return _mm256_set_pd(args...); } - template static auto SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept + template static register_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) setr(Args... args) noexcept { return _mm256_setr_pd(args...); } // comparison - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpeq(auto lhs, auto rhs) noexcept { return _ext256_cmpeq_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) cmpgt(auto lhs, auto rhs) noexcept { return _ext256_cmpgt_pd(lhs, rhs); } // conversion - static auto SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) expand(auto lhs, auto rhs) noexcept { return _mm256_cvtps_epi32(lhs, rhs); } @@ -6671,7 +6777,7 @@ template <> struct SimdImpl256 * @param index Selected lane in the range `[0, 4)`. * @return Selected scalar lane. */ - static double SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const __m256d lhs, const int index) noexcept + static double SIMD_FLAGS(In, RegisterOnly, ForceInline) extract_slow(const register_t lhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "64-bit floating-point extraction requires a valid 256-bit lane index"); const __m256i first_word = _mm256_set1_epi32(index * 2); @@ -6680,12 +6786,12 @@ template <> struct SimdImpl256 return SimdImpl128::template extract<0>(_mm_castsi128_pd(_mm256_castsi256_si128(selected))); } /** @brief Replaces the compile-time-selected 64-bit floating-point lane during constant evaluation. */ - template [[nodiscard]] constexpr static auto insert_constexpr(auto lhs, const double rhs) noexcept + template [[nodiscard]] constexpr static register_t insert_constexpr(auto lhs, const double rhs) noexcept { return register_insert_constexpr(lhs, rhs, static_cast(index)); } /** @brief Replaces the compile-time-selected 64-bit floating-point lane. */ - template static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const double rhs) noexcept + template static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert(auto lhs, const double rhs) noexcept { constexpr int half_index = index / 2; constexpr int lane_index = index % 2; @@ -6708,7 +6814,7 @@ template <> struct SimdImpl256 * @param index Selected lane index. * @return Register with the selected lane replaced. */ - static __m256d SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const __m256d lhs, const double rhs, const int index) noexcept + static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline) insert_slow(const register_t lhs, const double rhs, const int index) noexcept { SIMDLIB_PRECONDITION(index >= 0 && index < 4, "64-bit floating-point insertion requires a valid 256-bit lane index"); if (index < 2) @@ -6721,11 +6827,11 @@ template <> struct SimdImpl256 } // unpack / pack - static auto SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_lo(auto lhs, auto rhs) noexcept { return _mm256_unpacklo_pd(lhs, rhs); } - static auto SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) unpack_hi(auto lhs, auto rhs) noexcept { return _mm256_unpackhi_pd(lhs, rhs); } @@ -6737,7 +6843,7 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the shuffled lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) shuffle_slow(auto lhs, auto rhs, const int imm8) noexcept { return register_shuffle_double_slow(lhs, rhs, imm8); } @@ -6747,15 +6853,20 @@ template <> struct SimdImpl256 * @param imm8 Runtime control byte. * @return Register containing the selected lanes. */ - static auto SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept + static register_t SIMD_FLAGS(InOut, ForceInline) blend_slow(auto lhs, auto rhs, const int imm8) noexcept { - return register_blend_slow(lhs, rhs, static_cast(imm8)); + const __m256i selectors = make_static_register>([]() constexpr noexcept + { + return static_cast(1u << index); + }); + const __m256i selected = _mm256_and_si256(_mm256_set1_epi64x(imm8), selectors); + return _mm256_blendv_pd(lhs, rhs, _mm256_castsi256_pd(_mm256_cmpeq_epi64(selected, selectors))); } /** @brief Selects 64-bit floating-point lanes from two 256-bit registers with an immediate control. */ - template constexpr static auto SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const __m256d lhs, const __m256d rhs) noexcept + template constexpr static register_t SIMD_FLAGS(InOut, RegisterOnly, ForceInline, Flatten) blend(const register_t lhs, const register_t rhs) noexcept { if (std::is_constant_evaluated()) - return register_blend_slow(lhs, rhs, static_cast(imm8)); + return register_blend_constexpr(lhs, rhs, static_cast(imm8)); return _mm256_blend_pd(lhs, rhs, imm8 & 0x0F); } }; @@ -6894,7 +7005,7 @@ template struct SimdMappings<256, element_t> : public SimdImpl { if (std::is_constant_evaluated()) { - return register_from_array(data); + return register_from_array_constexpr(data); } else { @@ -7224,16 +7335,27 @@ template struct SimdMappings<256, element_t> : public SimdImpl /// Returns a mask of the most significant BIT of each element. static mask_t SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) movemask_slim(const vector_t lhs) noexcept { - if constexpr (std::is_integral_v) + if constexpr (std::is_integral_v && sizeof(element_t) == 1) + { + // Byte lanes already match VMOVMSK's native granularity, so no lane compaction is required. + return _mm256_movemask_epi8(lhs); + } + else if constexpr (std::is_integral_v && sizeof(element_t) == sizeof(float)) + { + // Reinterpret each integer lane as float so VMOVMSKPS reads the same lane sign bits without conversion. + return _mm256_movemask_ps(_mm256_castsi256_ps(lhs)); + } + else if constexpr (std::is_integral_v && sizeof(element_t) == sizeof(double)) + { + // Reinterpret each integer lane as double so VMOVMSKPD reads the same lane sign bits without conversion. + return _mm256_movemask_pd(_mm256_castsi256_pd(lhs)); + } + else if constexpr (std::is_integral_v) { // Note: _mm256_shuffle_epi8 is lane-local; compress the resulting byte-mask to element bits. const uint32_t raw = static_cast(movemask(swizzle_msb(lhs))); - if constexpr (sizeof(element_t) == 1) - { - return static_cast(raw); - } - else if constexpr (sizeof(element_t) == 2) + if constexpr (sizeof(element_t) == 2) { // 8 elements per 128-bit lane => bits 0..7 and 16..23 return static_cast((raw & 0x00FFu) | ((raw >> 8) & 0xFF00u)); } @@ -7279,30 +7401,18 @@ template struct SimdMappings<256, element_t> : public SimdImpl } /// Returns the shuffle order to move the most significant byte of each element into the least significant bytes. - constexpr static int_vector_t get_msb_swizzle_order() noexcept + constexpr static int_vector_t SIMD_FLAGS(Out, RegisterOnly, ForceInline, Flatten) get_msb_swizzle_order() noexcept { - int_vector_t seq{}; - constexpr const auto elem_size = sizeof(element_t); - constexpr const auto elems_per_lane = 16 / elem_size; - - for (std::size_t lane = 0; lane < 2; ++lane) + return make_static_register>([]() constexpr noexcept { - const std::size_t lane_base = lane * 16; - for (std::size_t i = 0; i < 16; ++i) - { - if (i < elems_per_lane) - { - // Select the MSB byte of each element within the 128-bit lane. - register_set_constexpr(seq, lane_base + i, static_cast((i * elem_size) + (elem_size - 1))); - } - else - { - // Zero out the rest (PSHUFB: high bit set => 0). - register_set_constexpr(seq, lane_base + i, 0x80); - } - } - } - return seq; + constexpr std::size_t element_size = sizeof(element_t); + constexpr std::size_t elements_per_group = 16 / element_size; + constexpr std::size_t group_index = index % 16; + if constexpr (group_index < elements_per_group) + return static_cast((group_index * element_size) + (element_size - 1)); + else + return std::uint8_t{0x80}; + }); } /// Swizzle the vector to only contain the most significant bit of each byte. diff --git a/include/SimdLib/RegisterMask.h b/include/SimdLib/RegisterMask.h index 0a32631..967106d 100644 --- a/include/SimdLib/RegisterMask.h +++ b/include/SimdLib/RegisterMask.h @@ -55,7 +55,13 @@ class RegisterMask final */ [[nodiscard]] constexpr bool SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) any(this RegisterMask value) noexcept { - return value.bits() != 0; + if (std::is_constant_evaluated()) + return value.bits() != 0; + + using byte_api_type = Api; + const auto bits = api_type::template bit_cast(value.native); + // Canonical predicates are all-zero or all-one per lane, so any set bit proves that a lane is true. + return byte_api_type::testz(bits, bits) == 0; } /** @@ -66,7 +72,29 @@ class RegisterMask final */ [[nodiscard]] constexpr bool SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) all(this RegisterMask value) noexcept { - return value.bits() == all_bits; + if consteval + { + return value.bits() == all_bits; + } + else + { +#if SIMDLIB_TARGET_X86 + if constexpr (sizeof(element_type) == 2) + { + // Canonical predicates are all-one or all-zero per lane, so containment by an all-one mask proves every lane true. + if constexpr (register_bits == 128) + return _mm_testc_si128(value.native, _mm_cmpeq_epi8(value.native, value.native)) != 0; + else + return _mm256_testc_si256(value.native, _mm256_cmpeq_epi8(value.native, value.native)) != 0; + } + else + { + return value.bits() == all_bits; + } +#else + return value.bits() == all_bits; +#endif + } } /** @@ -77,7 +105,13 @@ class RegisterMask final */ [[nodiscard]] constexpr bool SIMD_FLAGS(In, RegisterOnly, ForceInline, Flatten) none(this RegisterMask value) noexcept { - return value.bits() == 0; + if (std::is_constant_evaluated()) + return value.bits() == 0; + + using byte_api_type = Api; + const auto bits = api_type::template bit_cast(value.native); + // Testing the complete canonical predicate avoids materializing a compact scalar lane mask. + return byte_api_type::testz(bits, bits) != 0; } /** diff --git a/tests/Register.tests.cpp b/tests/Register.tests.cpp index 17d5450..b37cc5a 100644 --- a/tests/Register.tests.cpp +++ b/tests/Register.tests.cpp @@ -187,8 +187,11 @@ template void require_mask_contracts() REQUIRE(alternating.bits() == alternating_bits); REQUIRE(rewrapped.bits() == alternating_bits); REQUIRE(alternating.any()); + REQUIRE_FALSE(alternating.none()); REQUIRE_FALSE(alternating.all()); REQUIRE(all_true.bits() == all_bits); + REQUIRE(all_true.any()); + REQUIRE_FALSE(all_true.none()); REQUIRE(all_true.all()); REQUIRE((alternating | inverse).bits() == all_bits); REQUIRE((alternating & inverse).none()); diff --git a/tests/RegisterBasicOperations.tests.cpp b/tests/RegisterBasicOperations.tests.cpp index b246b36..2c3a653 100644 --- a/tests/RegisterBasicOperations.tests.cpp +++ b/tests/RegisterBasicOperations.tests.cpp @@ -524,8 +524,13 @@ void require_complete_register_shifts() using word_register = SimdLib::Register; constexpr std::array words{0x0123456789ABCDEFULL, 0xFEDCBA9876543210ULL}; const word_register word_value = word_register::from_array(words); - constexpr std::array bit_counts{std::numeric_limits::lowest(), -1, 0, 1, 63, 64, 65, 127, 128, 129, std::numeric_limits::max()}; - for (const int count : bit_counts) + for (int count = 0; count < 128; ++count) + { + REQUIRE(word_value.shift_bits_left_slow(count).to_array() == whole_left(words, count)); + REQUIRE(word_value.shift_bits_right_slow(count).to_array() == whole_right(words, count)); + } + constexpr std::array out_of_range_bit_counts{std::numeric_limits::lowest(), -17, -1, 128, 129, std::numeric_limits::max()}; + for (const int count : out_of_range_bit_counts) { REQUIRE(word_value.shift_bits_left_slow(count).to_array() == whole_left(words, count)); REQUIRE(word_value.shift_bits_right_slow(count).to_array() == whole_right(words, count)); @@ -573,6 +578,84 @@ template void require_bitwise_type() require_bitwise_operations(); } +/** @brief Verifies every lane-sign combination while independently varying all non-sign payload bits. */ +template + requires(std::is_integral_v && (sizeof(element_t) == 2 || sizeof(element_t) == 4 || sizeof(element_t) == 8)) +void require_exhaustive_lane_sign_bits() +{ + using register_type = SimdLib::Register; + using bits_type = bit_integer_t; + constexpr auto sign_bit = static_cast(bits_type{1} << (std::numeric_limits::digits - 1)); + constexpr auto payload_mask = static_cast(~sign_bit); + constexpr std::uint32_t combination_count = std::uint32_t{1} << register_type::lane_count; + + for (std::uint32_t expected = 0; expected < combination_count; ++expected) + { + std::array sparse_payload{}; + std::array dense_payload{}; + for (std::size_t lane = 0; lane < register_type::lane_count; ++lane) + { + const auto lane_sign = static_cast(((expected >> lane) & 1U) != 0 ? sign_bit : bits_type{}); + const auto sparse_bits = static_cast((bits_type{1} << (lane % (std::numeric_limits::digits - 1))) | bits_type{0x15}); + const auto dense_bits = static_cast(~static_cast(lane * 0x10203U + expected * 0x101U)); + sparse_payload[lane] = scalar_from_bits(static_cast(lane_sign | (sparse_bits & payload_mask))); + dense_payload[lane] = scalar_from_bits(static_cast(lane_sign | (dense_bits & payload_mask))); + } + + REQUIRE(register_type::from_array(sparse_payload).lane_sign_bits() == expected); + REQUIRE(register_type::from_array(dense_payload).lane_sign_bits() == expected); + } +} + +/** @brief Runs exhaustive 16-bit, 32-bit, and 64-bit integral sign-mask coverage at both supported widths. */ +template void require_exhaustive_lane_sign_bits_type() +{ + require_exhaustive_lane_sign_bits(); + if constexpr (SIMDLIB_REGISTER_TEST_ENABLE_256) + require_exhaustive_lane_sign_bits(); +} + +/** @brief Verifies representative byte-lane masks while independently varying every lane's non-sign payload. */ +template + requires(std::is_integral_v && sizeof(element_t) == 1) +void require_byte_lane_sign_bits() +{ + using register_type = SimdLib::Register; + constexpr std::uint32_t all_lane_bits = []() constexpr noexcept + { + if constexpr (register_type::lane_count == 32) + return std::numeric_limits::max(); + else + return (std::uint32_t{1} << register_type::lane_count) - 1; + }(); + constexpr std::array masks{std::uint32_t{}, all_lane_bits, std::uint32_t{1}, std::uint32_t{1} << (register_type::lane_count - 1), + std::uint32_t{0xA5A5A5A5} & all_lane_bits, std::uint32_t{0x3CC35AA5} & all_lane_bits}; + + for (const std::uint32_t expected : masks) + { + std::array sparse_payload{}; + std::array dense_payload{}; + for (std::size_t lane = 0; lane < register_type::lane_count; ++lane) + { + const auto sign = static_cast(((expected >> lane) & 1U) != 0 ? 0x80U : 0U); + sparse_payload[lane] = scalar_from_bits(static_cast(sign | ((lane * 5U + 1U) & 0x7FU))); + dense_payload[lane] = + scalar_from_bits(static_cast(sign | (0x7FU - static_cast((lane * 11U) & 0x7FU)))); + } + + REQUIRE(register_type::from_array(sparse_payload).lane_sign_bits() == expected); + REQUIRE(register_type::from_array(dense_payload).lane_sign_bits() == expected); + } +} + +/** @brief Runs byte-lane sign-mask coverage for signed and unsigned elements at both supported widths. */ +template void require_byte_lane_sign_bits_type() +{ + require_byte_lane_sign_bits(); + if constexpr (SIMDLIB_REGISTER_TEST_ENABLE_256) + require_byte_lane_sign_bits(); +} + /** @brief Runs per-lane shift coverage at both supported register widths. */ template void require_shift_type() { @@ -609,6 +692,22 @@ TEST_CASE("Register bitwise operations and sign masks preserve exact bits", "[si require_bitwise_type(); } +TEST_CASE("Register lane sign masks exhaustively ignore non-sign integer payload bits", "[simdlib][register][movemask][exhaustive]") +{ + require_exhaustive_lane_sign_bits_type(); + require_exhaustive_lane_sign_bits_type(); + require_exhaustive_lane_sign_bits_type(); + require_exhaustive_lane_sign_bits_type(); + require_exhaustive_lane_sign_bits_type(); + require_exhaustive_lane_sign_bits_type(); +} + +TEST_CASE("Register byte lane sign masks preserve lane order and ignore payload bits", "[simdlib][register][movemask][byte]") +{ + require_byte_lane_sign_bits_type(); + require_byte_lane_sign_bits_type(); +} + TEST_CASE("Register shifts match lane and complete-register boundary contracts", "[simdlib][register][shift]") { require_shift_type(); diff --git a/tests/codegen/RegisterCodegenFixture.h b/tests/codegen/RegisterCodegenFixture.h index a82e719..697c99d 100644 --- a/tests/codegen/RegisterCodegenFixture.h +++ b/tests/codegen/RegisterCodegenFixture.h @@ -26,6 +26,7 @@ using register_type = SimdLib::Register; #endif using uint_api_type = SimdLib::Api; using uint_native_type = typename uint_api_type::vector_t; +using byte_api_type = SimdLib::Api; #if SIMDLIB_CODEGEN_USE_WRAPPER using uint_register_type = SimdLib::Register; #endif @@ -117,7 +118,9 @@ SIMDLIB_CODEGEN_NOINLINE bool SIMD_FLAGS(In, RegisterOnly) simdlib_codegen_mask_ #if SIMDLIB_CODEGEN_USE_WRAPPER return SimdLibCodegen::register_type{lhs}.compare_equal(SimdLibCodegen::register_type{rhs}).any(); #else - return SimdLibCodegen::api_type::movemask_slim(SimdLibCodegen::api_type::compare_equal(lhs, rhs)) != 0; + const auto predicate = SimdLibCodegen::api_type::compare_equal(lhs, rhs); + const auto bits = SimdLibCodegen::api_type::template bit_cast(predicate); + return !SimdLibCodegen::byte_api_type::testz(bits, bits); #endif } diff --git a/tests/codegen/RegisterTypeMatrixCodegenFixture.h b/tests/codegen/RegisterTypeMatrixCodegenFixture.h index af4dd4f..01f9546 100644 --- a/tests/codegen/RegisterTypeMatrixCodegenFixture.h +++ b/tests/codegen/RegisterTypeMatrixCodegenFixture.h @@ -223,6 +223,41 @@ enum class scalar_operation extract_first, }; +#if !SIMDLIB_CODEGEN_USE_WRAPPER +/** @brief Mirrors the public mask any-lane Boolean-return boundary with the raw backend operation. */ +template [[nodiscard]] bool SIMD_FLAGS(In, ForceInline) raw_mask_any(native_t value) noexcept +{ + using api_type = api_t; + using byte_api_type = api_t; + const auto bits = api_type::template bit_cast(value); + return byte_api_type::testz(bits, bits) == 0; +} + +/** @brief Mirrors the public mask no-lanes Boolean-return boundary with the raw backend operation. */ +template [[nodiscard]] bool SIMD_FLAGS(In, ForceInline) raw_mask_none(native_t value) noexcept +{ + using api_type = api_t; + using byte_api_type = api_t; + const auto bits = api_type::template bit_cast(value); + return byte_api_type::testz(bits, bits) != 0; +} + +/** @brief Mirrors the public mask all-lanes Boolean-return boundary with the raw backend operation. */ +template [[nodiscard]] bool SIMD_FLAGS(In, ForceInline) raw_mask_all(native_t value) noexcept +{ + using api_type = api_t; + if constexpr (sizeof(element_t) == 2) + { + // Match RegisterMask::all(): canonical 16-bit predicates permit a whole-register containment test. + return api_type::test(value, api_type::compare_equal(value, value)) != 0; + } + else + { + return api_type::movemask_slim(value) == all_lane_bits(); + } +} +#endif + /** * @brief Emits one isolated scalar-result operation for exact wrapper/raw comparison. * @tparam operation Operation selected at compile time. @@ -231,7 +266,7 @@ enum class scalar_operation * @return Compact scalar result of the selected operation. */ template -[[nodiscard]] typename api_t::mask_t SIMD_FLAGS(In, ForceInline) scalar_result(native_t lhs, native_t rhs) noexcept +[[nodiscard]] auto SIMD_FLAGS(In, ForceInline) scalar_result(native_t lhs, native_t rhs) noexcept { using api_type = api_t; using register_type [[maybe_unused]] = register_t; @@ -247,11 +282,11 @@ template else if constexpr (operation == scalar_operation::mask_bits) return mask.bits(); else if constexpr (operation == scalar_operation::mask_any) - return static_cast(mask.any()); + return mask.any(); else if constexpr (operation == scalar_operation::mask_all) return static_cast(mask.all()); else if constexpr (operation == scalar_operation::mask_none) - return static_cast(mask.none()); + return mask.none(); else if constexpr (operation == scalar_operation::equal) return static_cast(left == right); else if constexpr (operation == scalar_operation::not_equal) @@ -264,15 +299,15 @@ template else if constexpr (operation == scalar_operation::lane_sign_bits || operation == scalar_operation::mask_bits) return api_type::movemask_slim(lhs); else if constexpr (operation == scalar_operation::mask_any) - return static_cast(api_type::movemask_slim(lhs) != 0); + return raw_mask_any(lhs); else if constexpr (operation == scalar_operation::mask_all) - return static_cast(api_type::movemask_slim(lhs) == all_lane_bits()); + return static_cast(raw_mask_all(lhs)); else if constexpr (operation == scalar_operation::mask_none) - return static_cast(api_type::movemask_slim(lhs) == 0); + return raw_mask_none(lhs); else if constexpr (operation == scalar_operation::equal) - return static_cast(api_type::movemask_slim(api_type::compare_equal(lhs, rhs)) == all_lane_bits()); + return static_cast(raw_mask_all(api_type::compare_equal(lhs, rhs))); else if constexpr (operation == scalar_operation::not_equal) - return static_cast(api_type::movemask_slim(api_type::compare_equal(lhs, rhs)) != all_lane_bits()); + return static_cast(!raw_mask_all(api_type::compare_equal(lhs, rhs))); else return static_cast(api_type::template extract<0>(lhs)); #endif @@ -388,6 +423,14 @@ native_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) from_lanes(const return SimdLibTypeMatrixCodegen::scalar_result(lhs, rhs); \ } +#define SIMDLIB_DEFINE_TYPE_MATRIX_BOOLEAN(token, element_type, operation) \ + /** @brief Compares one isolated Boolean-result operation with its raw Api expression. */ \ + SIMDLIB_TYPE_MATRIX_NOINLINE bool SIMD_FLAGS(In) simdlib_type_matrix_##operation##_##token( \ + SimdLibTypeMatrixCodegen::native_t lhs, SimdLibTypeMatrixCodegen::native_t rhs) noexcept \ + { \ + return SimdLibTypeMatrixCodegen::scalar_result(lhs, rhs); \ + } + #define SIMDLIB_DEFINE_TYPE_MATRIX_COMMON_FIXTURES(token, element_type) \ SIMDLIB_DEFINE_TYPE_MATRIX_VECTOR(token, element_type, zero) \ SIMDLIB_DEFINE_TYPE_MATRIX_VECTOR(token, element_type, broadcast) \ @@ -415,9 +458,9 @@ native_t SIMD_FLAGS(Out, RegisterOnly, ForceInline) from_lanes(const SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, movemask) \ SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, lane_sign_bits) \ SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, mask_bits) \ - SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, mask_any) \ + SIMDLIB_DEFINE_TYPE_MATRIX_BOOLEAN(token, element_type, mask_any) \ SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, mask_all) \ - SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, mask_none) \ + SIMDLIB_DEFINE_TYPE_MATRIX_BOOLEAN(token, element_type, mask_none) \ SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, equal) \ SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, not_equal) \ SIMDLIB_DEFINE_TYPE_MATRIX_SCALAR(token, element_type, extract_first) \ @@ -512,6 +555,7 @@ static_assert(!SimdLib::IRegister::ShiftRight()); static_assert(construction_contract<128, float>()); static_assert(construction_contract<128, double>()); static_assert(setr_64bit_construction_contract()); +static_assert(static_register_construction_contract>()); static_assert(comparison_contract<128, std::int8_t>()); static_assert(comparison_contract<128, std::uint8_t>()); diff --git a/tests/constexpr/Api256Constexpr.tests.cpp b/tests/constexpr/Api256Constexpr.tests.cpp index dd528a4..61ea762 100644 --- a/tests/constexpr/Api256Constexpr.tests.cpp +++ b/tests/constexpr/Api256Constexpr.tests.cpp @@ -23,6 +23,7 @@ static_assert(construction_contract<256, std::int64_t>()); static_assert(construction_contract<256, std::uint64_t>()); static_assert(construction_contract<256, float>()); static_assert(construction_contract<256, double>()); +static_assert(static_register_construction_contract>()); static_assert(comparison_contract<256, std::int8_t>()); static_assert(comparison_contract<256, std::uint8_t>()); diff --git a/tests/constexpr/ApiConstexprContracts.h b/tests/constexpr/ApiConstexprContracts.h index 2f99a18..02b7bee 100644 --- a/tests/constexpr/ApiConstexprContracts.h +++ b/tests/constexpr/ApiConstexprContracts.h @@ -96,7 +96,7 @@ template [[nodiscard]] consteval bool detail_ { using simd = Api; constexpr auto values = lane_values(); - auto value = Detail::register_from_array(values); + auto value = Detail::register_from_array_constexpr(values); for (std::size_t index = 0; index < simd::element_count; ++index) { @@ -175,6 +175,23 @@ template [[nodiscard]] consteval bool constru return simd::extract_slow(replaced, static_cast(simd::element_count - 1)) == replacement; } +/** + * @brief Verifies generator-based native register construction during constant evaluation. + * @tparam Implementation Native byte backend selected by the owning width-specific probe. + * @return True when every generated lane preserves its compile-time index-derived value. + */ +template [[nodiscard]] consteval bool static_register_construction_contract() noexcept +{ + constexpr auto value = Detail::make_static_register([]() constexpr noexcept + { + return static_cast((index * 7U) + 3U); + }); + std::array expected{}; + for (std::size_t index = 0; index < expected.size(); ++index) + expected[index] = static_cast((index * 7U) + 3U); + return Detail::register_to_array(value) == expected; +} + /** * @brief Verifies signed and unsigned 64-bit forward-order construction during constant evaluation. * @return True when lane order and complete unsigned bit patterns are preserved.