[dgarske]

Description

Supersedes #11808 (incorporating work by @armpribb).

Fixes undefined behavior (UBSan) in sp_c32.c, sp_c64.c, and sp_int.c when compiling without -fwrapv (e.g., in CMake, IDE, and vendor builds; Autotools masked this by defaulting to -fwrapv via hardening flags). The new regression test then found a wrong result in the ARM32 and Cortex-M SP assembly, fixed here as well.

Issues Addressed

• Signed left shifts: In P-256 and P-384 Montgomery normalizers.
• Signed overflow: In limb subtraction for 2048-bit Toom-3 and 3072-bit Karatsuba multipliers.
• Signed multiply: In de Bruijn bit-count (num_bits) helpers.
• Null-pointer offset: In sp_to_unsigned_bin_at_pos() before validating out.
• Dropped carry (ARM32 / Thumb2 assembly): In the unrolled squares sp_3072_sqr_12, sp_384_sqr_12, sp_521_sqr_17 and sp_1024_sqr_16, the first product of a column is added twice into a three-word accumulator, and the mov that clears the stale top word was emitted in both steps, so the second step discarded the carry of the first. Needs a run of limbs at 2^32 - 1 in the four lowest or four highest columns; sp_RsaPublic_3072() with m = 2^3072 - 189 hit it. The 8-limb squares use UMAAL/UMLAL and were correct.

Commits

1. wolfcrypt: avoid signed overflow in ML-DSA and C SP arithmetic (@armpribb)
   Adds core fixes and regression tests (RSA modexp near-max moduli, P-256 sign/verify, and ML-DSA-65 public-key digest).
2. test: respect digest limits in P-256 SP regression (@armpribb)
   Skips P-256 tests in digest-restricted configurations.
3. SP C: regenerate with unsigned limb packing, subtraction and bit counting
   Systematic fix applied directly to the generator (wolfSSL/scripts#710) and regenerated sp_c32.c / sp_c64.c. Uses unsigned arithmetic across all limb subtractions, normalizers, and bit-counting helpers.
4. Check the output buffer before offsetting it in sp_to_unsigned_bin_at_pos
   Validates a and out before pointer arithmetic, returning MP_VAL on NULL.
5. SP ARM32 and Thumb2: keep the carry of the first doubled product in each square column
   Generator fix in the same wolfSSL/scripts#710 and regenerated sp_arm32.c / sp_cortexm.c; the only change is the removed second mov in the four squares above.

Testing

Tested with Clang 18 using -fsanitize=undefined -fno-sanitize-recover=all -fno-wrapv:

• Word sizes: C32 and C64 (with/without SAKKE/ECCSI; --disable-sp --enable-sp-math-all).
• Result: testwolfcrypt passes cleanly. (Master previously aborted in sp_256_mod_mul_norm_5, sp_2048_sub_24, and sp_int.c.)
• Additional checks: GCC (--enable-all --disable-asm), make check (C SP backend), and check-source-text all pass.
• ARM32 and Cortex-M: --enable-sp-asm builds under qemu-arm, ARM32 with the armel-o0 CI flags (-marm -DWOLFSSL_SP_ARM_ARCH=6 -O0) and Cortex-M with -mthumb, both with SP P-384/P-521/SAKKE. Before the last commit the four squares disagree with the multiplies for all-0xffffffff and all-0xfffffffe operands and rsa_sp_modexp_test() fails at 3072 bits; after it they agree and testwolfcrypt passes. x86_64 and AArch64 assembly pass the new test unchanged.

[github-actions]

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m3

• FLASH: .text -40 B (-0.0%, 131,721 B / 262,144 B, total: 50% used)

gcc-arm-cortex-m4-baremetal

• FLASH: .text -64 B (-0.1%, 73,463 B / 262,144 B, total: 28% used)

gcc-arm-cortex-m4-min-ecc

• FLASH: .text -64 B (-0.1%, 68,057 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-sp-math

• FLASH: .text -64 B (-0.1%, 68,057 B / 262,144 B, total: 26% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text -64 B (-0.0%, 132,497 B / 262,144 B, total: 51% used)

stm32-sim-stm32h753

• FLASH: .text +328 B (+0.2%, 204,404 B / 2,097,152 B, total: 10% used)

zephyr-frdm_rw612

• FLASH: log_const_area -8 B, rodata +4 B, text +2,352 B (+0.8%, 284,812 B / 67,108,864 B, total: 0% used)

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #12053

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs
Coverage: 4 of 4 in-scope changed file(s) opened by the reviewer

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Max