ML-DSA on x86-64: polynomial arithmetic in SSE2 (sign −19%, verify −11% instructions) - #467
Merged
Merged
Conversation
vg_mldsa_ntt and vg_mldsa_inv_ntt now compute on four coefficients at a
time in SSE2 registers, as ML-KEM's x86-64 NTT does: a Montgomery
multiplication with pmuludq (the even doublewords, then the odd ones moved
down by pshufd), conditional additions of q with psrad masks, and for the
layers with len 2 and 1 the coefficients of two or four blocks gathered with
punpck{l,h}qdq (and pshufd) and interleaved back. The zetas are a table in
Montgomery form that the prologue stores in scratch; the multiplications run
inside ML-KEM's withMxcsr, so Intel's MCDT mitigation holds.
Proofs: the lanes' arithmetic (VArith), the butterflies on registers
(VLanes), the loads and stores of four coefficients and the zetas (VMem),
the layers (VLay, VLay21), and the functions (Ntt, NttInv). Signing and
verification now check that their code loads MXCSR only to restore it
(ctlOk, through a compositional ctlC for verification) rather than never,
since their primitives now do.
ML-DSA-65 on this machine: sign 1.53 ms -> 0.81 ms, verify 281 us ->
181 us, keygen 276 us -> 248 us.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ddof3szoTi7HB8iCsM2MCr
vg_mldsa_multiply_ntt, vg_mldsa_multiply_add_ntt, vg_mldsa_add and vg_mldsa_sub now compute on four coefficients at a time in SSE2 registers, with the vector helpers of the NTT (Vec.lean). The products are two Montgomery multiplications: f·g·2⁻³², then by 2⁶⁴ mod q, which is f·g in [0, 2q), reduced with vcsub (for multiply_add, h is then added and the sum reduced). They use pmuludq, so they run inside ML-KEM's withMxcsr. These functions have no working space and use no stack, so MXCSR goes through the last 8 bytes of h, addressed through r8 = h. The last four coefficients of h are loaded into xmm6 first. The loop stores the first 252 coefficients. The last four are computed from registers before MXCSR is loaded back, and stored after it. Their callers' proofs are unchanged: the functions still need no stack and never write rsp. add and sub use paddd/psubd and vcsub/vcadd. Proofs: the lanes of a product (mul_lane, mulAdd_lane), the loop (Mul.step, Mul.loop_ok), the last block (Mul.last), the function (Mul.fn_ok), and withMxcsr through the end of a polynomial (withMxcsrH_ok). AddSub is reproven the same way. ML-DSA-65 instructions executed (callgrind, PR #464 -> this): sign -19%, verify -11%, keygen -2%. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ddof3szoTi7HB8iCsM2MCr
…into claude/fervent-einstein-ukl7t7-mul
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ddof3szoTi7HB8iCsM2MCr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #464 (the SSE2 NTT). Its base is that PR's branch, so the diff here shows only this change. Once #464 merges, I'll retarget this PR to
main.What
vg_mldsa_multiply_ntt,vg_mldsa_multiply_add_ntt,vg_mldsa_addandvg_mldsa_subon x86-64 now process four coefficients at a time in SSE2 registers. They reuse the vector helpers from #464 (Impl/MlDsa/X86_64/Arith/Vec.lean).pmuludqon even lanes andpshufd 0xF5for odd lanes. The first givesf·g·2⁻³²; multiplying that by2⁶⁴ mod qgivesf·gin[0, 2q), whichvcsubreduces.multiply_addthen addshand reduces again.paddd/psubd, thenvcsub/vcadd.MXCSR without stack
Because of
pmuludq, the multiplications have to run inside ML-KEM'swithMxcsr, so Intel's MCDT mitigation holds (ci/check_mcdt.pypasses). These functions have no scratch buffer, and I didn't want to give them a stack frame either. On x86-64,Exec.frameSpand the taint analysis don't support frames yet, and every ML-DSA caller relies onNoSp. So MXCSR goes through the last 8 bytes ofh:r8 = hand load the last four coefficients ofhintoxmm6.withMxcsr r8 1016 (prologue; 63-iteration loop over the first 252 coefficients; the last four from registers).The contracts are unchanged (no stack), and so are the proofs of sign, keygen and verify.
Proofs (untrusted)
mul_lane/mulAdd_lane: one lane of the product, from the existingmont_mont_R2.Mul.step,Mul.loop_ok,Mul.last,Mul.fn_ok: the loop, the last block, and the whole function.withMxcsrH_ok(Proof/MlDsa/X86_64/Arith/Mxcsr.lean):withMxcsrthrough the end of a writable polynomial, adapted from ML-KEM'swithMxcsr_ok.AddSubis reproven the same way.taint_decide, as before.There are no changes to TCB or Spec.
Performance
ML-DSA-65 instructions executed (callgrind, 20 operations, #464 → this PR):
Criterion timings on this shared VM were within noise (±10%), so I'm reporting instruction counts.
docs/algorithms/ml-dsa-*.tomlnow say "SSE2 polynomial arithmetic", and the README tables are regenerated.Checks run
lake buildlake env lean --run Emit.lean --checkcheck_lean_imports,check_lean_speed,check_vectors,check_arch_gates,check_variants,check_mcdtcargo fmt --check,cargo clippy --all-targets -D warningscargo test --releasewith Wycheproof: all pass🤖 Generated with Claude Code
https://claude.ai/code/session_01Ddof3szoTi7HB8iCsM2MCr
Generated by Claude Code