liboqs Fails Depending on the OpenSSL Provider Version Used

[ashman-p]

An active OpenSSL FIPS Provider can be based on a lower version of OpenSSL than the version of libcrypto currently loaded.
e.g. OpenSSL version could be 3.3 while the FIPS Provider could based on 3.0.

In this scenario, I came across the case where liboqs use of OpenSSL's SHA3 APIs failed.
SHA3_shake256_x4_inc_squeeze(), SHA3_shake128_x4_inc_squeeze failed because EVP_DigestSqueeze is not supported in OpenSSL 3.0.

The question therefore is, how can this be handled in general? I think it would be good to keep liboqs provider agnostic if possible.

[baentsch]

I think it would be good to keep liboqs provider agnostic if possible.

We could make this a goal -- imo it is not at this time: If someone activates a downlevel FIPS provider, it's permitted and may lead to failures such as this. Thinking this through now, my suggestion would be to keep this behaviour (possibly calling possible problems out by way of documentation) assuming there's got to be a reason for someone configuring libcrypto this way.

Of course, liboqs could prohibit such behaviour by propq-forcing load of the respective openssls "default" provider to avoid such problems (and checking for the OpenSSL version as to whether EVP_DigestSqueeze is supported at all).

Which way would be your preference to go @ashman-p ? Being completely provider agnostic IMO is impossible as providers are a core mechanism in OpenSSL >= 3.0. But of course I may misunderstand what you mean by being "provider agnostic"...

[ashman-p]

@baentsch, consider the info below.

The current code looks like this ...

#if OPENSSL_VERSION_NUMBER >= 0x30300000L
      USE OSSL_METHOD()
#else
      USE_OQS_METHOD()
#endif

A solution that I think could work is to modify the existing code with ...

#if OPENSSL_VERSION_NUMBER >= 0x30300000L
        rc = USE OSSL_METHOD()
        if (rc == EVP_R_METHOD_NOT_SUPPORTED)  
               USE_OQS_METHOD()
#else
      USE_OQS_METHOD()
#endif

Let me know what you think.

[baentsch]

That looks like a doable way -- under the assumption that all functions thus guarded a) can be invoked "effect-free" on the parameters (i.e., can be run a second time with the OQS_METHOD) and b) all have a unique rc that can be tested (EVP_R_METHOD_NOT_SUPPORTED): Is this both the case, @ashman-p ?

[ashman-p]

Thanks @baentsch, I will need to investigate. This was a potential way to reduce coupling liboqs to the provider layer. I will investigate more to bottom this out. Thanks.