Openssl-3.0-fips hash + sign

[lrstewart]

Problem:

Openssl-3.0-fips does not support hashing with MD5 (see #5045). Openssl-3.0-fips also doesn't support our use of EVP_MD_CTX_set_pkey_ctx (see #5047).

Solution

There are currently two ways to calculate a hash in s2n-tls: the "legacy" / "low-level" way that uses legacy, non-FIPS methods like SHA1_hash(), and the modern EVP way that uses methods like EVP_Digest(). The EVP methods are preferred, but currently only used for FIPS.

There are also currently two ways to sign: the legacy way that uses non-FIPS methods like ECDSA_sign(), and the modern EVP way that uses methods like EVP_DigestSign(). The EVP methods are preferred, but currently only used for FIPS. openssl-1.0.2-fips can't use our current EVP logic due to some missing methods (EVP_MD_CTX_set_pkey_ctx and EVP_MD5_SHA1). The EVP signing method requires an EVP hash, but the legacy signing method can use either a legacy or EVP hash.

There will need to be a third way to sign to support openssl-3-fips. That would be to sign using EVP without EVP_MD_CTX_set_pkey_ctx, using EVP_pkey_sign(). However, that would require us to use an externally calculated hash, which wouldn't be allowed by FIPS 140-3. That would mean that we could use a legacy hash though. We can call the two ways of signing with EVP "EVP" and "EVP-FIPS-140-3".

This is the current state of our library:

Libcrypto	Hash method	Signing Method
openssl-1.0.2	Legacy	Legacy
openssl-1.0.2-fips	EVP	Legacy
awslc-fips	EVP	EVP-FIPS-140-3
openssl-3-fips	n/a	n/a
other	Legacy	Legacy

This is where I believe the library should be:

Libcrypto	Hash method	Signing Method
openssl-1.0.2	EVP	EVP
openssl-1.0.2-fips	n/a	n/a
awslc-fips	EVP	EVP-FIPS-140-3
openssl-3-fips	EVP	EVP
other	EVP	EVP

Previously, I had openssl-1.0.2 and openssl-3-fips using "EVP+Legacy", referring to using EVP for all algorithms except MD5 / MD5+SHA1. That was based on the assumption that we'd have to use the Legacy method for hashes in openssl-3.0-fips because the fips provider doesn't include md5. Unfortunately, we need EVP support for md5 because when signing, RSA PKCS1 needs to know the hash algorithm. That means we already have to load a provider that supports MD5, like the Default provider. If we're going to load that provider anyway, we may as well always use EVP hashing.

Next steps

In pursuit of the above end result, I need to:

• 1. Cleanup. Remove the special casing for openssl-1.0.2-fips to use the EVP APIs with MD5+SHA1. That code is not being used anymore. refactor: remove unused evp support for md5+sha1 #5106 No longer needed. Revert "refactor: remove unused evp support for md5+sha1 (#5106)" #5118
• 2. Switch to EVP hashing
  • 2a. Switch EVP logic to use explicit fetch refactor: use EVP_MD_fetch() if available #5116
  • 2b. Switch all libcryptos to use EVP hashing refactor: always use EVP hashing #5121
  • 2c. (not a blocker) Clean up the legacy hash refactor: always use EVP hashing #5121
• 3. Switch to EVP signing
  • 3a. Add the new EVP signing method refactor: add alternative EVP signing method #5141
  • 3b. Switch all libcryptos except awslc-fips to use the new EVP signing method refactor: add alternative EVP signing method #5141
  • 3c. (not a blocker) Clean up all the legacy pkeys Completely replace legacy pkey impls with EVP impl #5149