sp x86_64: separate lane selection from vector-register ownership

[kaleb-himes]

Description

Pairs with: https://github.com/wolfSSL/scripts/pull/677

• CPUID alone picks the lane; the chosen lane saves the vector registers.
  This is a CPU step, not a fallback.
• A refused save returns the error. The other lane never runs.
• Every SP mulmod entry point saves for itself, both lanes (the non-AVX2
  table lookups are SSE2). P-1024 base lane excepted, it has none.
• cpuid_select_flags / cpuid_set_flag / cpuid_clear_flag re-run the power-on
  self test in FIPS builds, which resets every CAST to INIT, then calls
  wc_RunAllCast_fips() so the new lanes are CAST tested immediately rather
  than on first use. This is what the kernel module already does at load.
• The SP x86_64 white-box test now fails if a refused save still lets a call
  run, so a regression is caught in tree.

Behavior change: on AVX2-capable x86_64, SP RSA/ECC entry points can
return the vector-register save error instead of silently running the
non-AVX2 lane. Userspace never refuses the save.

Testing

20 test cells, all passing.

Builds and gates

• FIPS-ready, FIPS v7, and FIPS v7 with the SP assembly lanes on, full test
  suite on each.
• A build with AVX2 compiled out entirely.
• The fail-closed dev build under strict warnings.
• sp_x86_64.c reproduces byte for byte from the generator, with a control
  proving the pre-change file still differs.
• Windows: the whole file sits inside one feature guard, the Windows FIPS
  settings enable none of it, and with those settings it compiles to zero
  symbols. Windows gets no code from this change.

Refused saves must fail, never switch lanes

• Forced save failures into ECC P-256, P-384, P-521 and RSA 2048 and 3072,
  in a non-FIPS build and again against the real FIPS v7 module. Every call
  fails; results after the injection stops match those from before it, so
  the fixed point cache is not left half built.
• 8 threads at once on one shared cache entry, half clean, a quarter always
  failing, a quarter alternating: 750 clean results and 450 refusals, every
  clean result matching the reference. Non-FIPS and FIPS.

CPU feature change

• Userspace: clearing and setting AVX2 re-runs the self test and the CASTs.
  Proven with a witness algorithm the test program never calls, which goes
  from never run to passed purely because of the flag change.
• In kernel: same thing driven from inside the module, both directions, with
  the module healthy afterwards.

Real kernels

• Local KVM on an Intel Core Ultra 9 285K, Linux 6.16.12: the FIPS module
  with the in-kernel crypto test, booted twice, once with AVX2 visible and
  once with it hidden so the non-AVX2 lanes run. Both pass the power-on self
  test, pass the in-kernel test, re-verify at unload and unload clean, with
  no kernel warnings.
• Same guest with a save fuzzer: 8 loads, every one met a refused save,
  every one reported the error, no crash and no kernel warning.
• AWS spot on real silicon against the stock Amazon Linux kernel 6.12.103:
  AMD EPYC and Intel Xeon, each running the FIPS test suite, the feature
  change check, the forced failure check, and the FIPS kernel module loading
  with the power-on self test and the in-kernel test, then unloading clean.
• AWS spot Graviton, aarch64: the cpuid change is in the part of cpuid.c
  shared by Intel, aarch64, 32-bit Arm and PowerPC, so the FIPS build with
  the Arm assembly lanes runs the test suite and the feature change check
  there too.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-pq

• FLASH: .text +448 B (+0.1%, 303,048 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +448 B (+0.1%, 304,008 B / 1,048,576 B, total: 29% used)

linuxkm-standard

• Data: __patchable_function_entries +40 B (+0.1%, 49,936 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11422

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-src, wolfssl-bugs

Findings: 4
4 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

[kaleb-himes]

retest this please

Dismiss to re-request

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11422

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-src, wolfssl-bugs
Findings: 1

Required changes (1)

Small-memory signing configuration no longer compiles

File: wolfcrypt/src/wc_mldsa.c:10447
Function: mldsa_sign_with_seed_mu
Category: API contract violations

mldsa_sign_with_seed_mu() calls the new four-argument mldsa_vec_check_low() with two arguments under WOLFSSL_MLDSA_SIGN_SMALL_MEM and WOLFSSL_MLDSA_SIGN_CHECK_W0. Known #6771 instead concerns signing conformance.

Related known finding #6771 (similar but distinct): Both concern optional early-rejection checks in mldsa_sign_with_seed_mu, but this is a mismatched helper-call signature causing a compile failure, while #6771 is a FIPS-conformance deviation caused by the checks themselves; their fixes differ.

Suggested fix: Pass the vector length and &valid, assigning the helper's return value to ret as at the other updated call sites.
Basis: ISO/IEC 9899:2018 §6.5.2.2 requires function-call arguments to agree with the visible prototype.

Referenced code: wolfcrypt/src/wc_mldsa.c:10447-10448 (2 lines)

---

This review was generated automatically by Fenrir. Reported findings require changes before merge.