Building on macOS as generic Unix, without dependency on SecureTransport

[barracuda156]

Describe the feature

This does not need to be fixed here, but suggestions are greatly appreciated.

Currently the library unconditionally uses SecureTransport on macOS. However, SecureTransport is not supported in older macOS versions (or may not have required features implemented).

What is a suitable fallback in such a case? I know how to add code conditionally, but I am not sure whether existing posix code gonna work correctly and which settings are to be chosen.

I can build aws-c-io without SecureTransport with minimal patches: https://github.com/macos-powerpc/powerpc-ports/tree/8db8b66eb3ea4a2d73aaf16822fe410747669161/devel/aws-c-io/files
If those are okay, then all good. If not, if will be very helpful to know what to change.

Use Case

This is needed to have AWS libs working on legacy macOS versions. (Yes, a niche case, and I am fine to carry patches in my repo, if it is not acceptable to add them here.)
It may be also useful for testing, as a non-default (possibly developer) option, on modern macOS.

Proposed Solution

Fall back to generic POSIX when SecureTransport-based implementation is unsupported.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[sfod]

From what I see in the patches, you switch implementation on apple platforms to BSD + POSIX code.

What is a suitable fallback in such a case? I know how to add code conditionally, but I am not sure whether existing posix code gonna work correctly and which settings are to be chosen.

It's hard to say without testing. These "legacy macOS versions" you mention, which specific versions you want to support?

Also, I see that you've implemented this patch a few months ago. Did you verify whether the patched version work in your environment?

[barracuda156]

@sfod Thank you for responding!

These "legacy macOS versions" you mention, which specific versions you want to support?

10.6.8 primarily, 10.5.8 if possible (the latter not strictly required).

Did you verify whether the patched version work in your environment?

Is there a decisive test case for this?

I added AWS ports to fix awscli2 – for a specific use case, and that did not work for some unrelated reason (I needed to access Linode bucket, accessing itself worked, but file upload did not). I am sure it was unrelated, since I had identical outcomes with pure-python awscli and with both on modern macOS. But since that did not work so far, I am not sure if this library works correctly either.

[sfod]

Yeah, 10.6.8 is too old, we can't access to it unfortunately.

Is there a decisive test case for this?

By verifying I meant trying to use the library in some app, like you tried to do with awscli.
This repo has tests. You can try to run the tcp_socket_communication test - if it succeeds then probably everything is fine as this test covers the event loop and socket functionality, which your patch touched.

[barracuda156]

Yeah, 10.6.8 is too old, we can't access to it unfortunately.

I don’t think it is needed, TBH. If someone from the upstream side could ensure aws-c-io builds correctly against openssl on any macOS, that will solve the problem.

[sfod]

I quickly checked your patch on macOS 26. kqueue and POSIX sockets work just fine, which is expected because they were actually available until recently. But some TLS tests have failed with s2n and openssl, and this raises concerns. So I wouldn't recommend you using this in production.

Saying that, I agree that giving users a way to change the underlying implementations may be beneficial. I'll bring this idea to the team. However, I can't give you any timelines considering this involves security.

[barracuda156]

Saying that, I agree that giving users a way to change the underlying implementations may be beneficial. I'll bring this idea to the team. However, I can't give you any timelines considering this involves security.

Thank you very much, greatly appreciated.

[barracuda156]

@sfod A quick update on the matter:

1. I can confirm that quite a number of TLS-related tests fail (most of those, but not all). A few other tests fail, but largely only TLS is the problem. On powerpc 76% of tests passed for me:

76% tests passed, 47 tests failed out of 192

Total Test time (real) = 200.60 sec

The following tests FAILED:
     38 - tls_client_channel_negotiation_success_mtls_tls12 (Failed)
     39 - tls_client_channel_negotiation_success_mtls_tls13 (Failed)
     87 - default_pki_path_exists (Failed)
     88 - tls_client_channel_negotiation_error_expired (Failed)
     89 - tls_client_channel_negotiation_error_wrong_host (Failed)
     90 - tls_client_channel_negotiation_error_wrong_host_with_ca_override (Failed)
     91 - tls_client_channel_negotiation_error_self_signed (Failed)
     92 - tls_client_channel_negotiation_error_untrusted_root (Failed)
     93 - tls_client_channel_negotiation_error_untrusted_root_due_to_ca_override (Failed)
     94 - tls_client_channel_negotiation_no_verify_expired (Failed)
     95 - tls_client_channel_negotiation_no_verify_wrong_host (Failed)
     96 - tls_client_channel_negotiation_no_verify_self_signed (Failed)
     97 - tls_client_channel_negotiation_no_verify_untrusted_root (Failed)
     98 - tls_client_channel_negotiation_error_broken_crypto_rc4 (Failed)
     99 - tls_client_channel_negotiation_error_broken_crypto_rc4_md5 (Failed)
    100 - tls_client_channel_negotiation_error_broken_crypto_dh480 (Failed)
    101 - tls_client_channel_negotiation_error_broken_crypto_dh512 (Failed)
    102 - tls_client_channel_negotiation_error_broken_crypto_null (Failed)
    103 - tls_client_channel_negotiation_error_tls13_to_tls12_server (Failed)
    104 - tls_client_channel_negotiation_error_override_legacy_crypto_tls11 (Failed)
    105 - tls_client_channel_negotiation_error_no_subject (Failed)
    106 - tls_client_channel_negotiation_success_no_verify_no_subject (Failed)
    107 - tls_client_channel_negotiation_error_no_common_name (Failed)
    108 - tls_client_channel_negotiation_success_no_verify_no_common_name (Failed)
    109 - tls_client_channel_negotiation_success_no_verify_incomplete_chain (Failed)
    110 - tls_client_channel_negotiation_success_tls12 (Failed)
    111 - tls_client_channel_negotiation_success_sha256 (Failed)
    112 - tls_client_channel_negotiation_success_rsa2048 (Failed)
    113 - tls_client_channel_negotiation_success_ecc256 (Failed)
    114 - tls_client_channel_negotiation_success_ecc384 (Failed)
    115 - tls_client_channel_negotiation_success_mozilla_modern (Failed)
    116 - test_duplicate_cert_import (Subprocess aborted)
    120 - tls_client_channel_negotiation_success (Failed)
    122 - tls_client_channel_no_verify (Failed)
    126 - test_rsa_2048_cert_import (Failed)
    127 - test_rsa_4096_cert_import (Failed)
    128 - test_ecc_p256_cert_import (Failed)
    129 - test_ecc_p384_cert_import (Failed)
    130 - test_ecc_p521_cert_import (Failed)
    131 - test_tls_cipher_preference (Failed)
    132 - test_concurrent_cert_import (Subprocess aborted)
    133 - test_pkcs8_import (Failed)
    134 - test_pkcs12_import (Failed)
    148 - future_wait_timeout_max (Failed)
    178 - shared_library_open_success (Failed)
    179 - shared_library_find_function_failure (Failed)
    180 - shared_library_find_function_success (Failed)
Errors while running CTest

Here two cleanup_in_* tests were disabled since they seem to hang, at least the first of them.

2. However, the issue seems unrelated to – or at least not primarily caused by – my patches. I have dropped my patches to aws-c-io and aws-c-cal, rebuilt them and ran tests on Catalina on x86_64:

82% tests passed, 34 tests failed out of 193

Total Test time (real) =  90.88 sec

The following tests did not run:
	 23 - test_socket_with_bind_to_interface (Skipped)

The following tests FAILED:
	 40 - tls_client_channel_negotiation_success_mtls_tls12 (Failed)
	 41 - tls_client_channel_negotiation_success_mtls_tls13 (Failed)
	 89 - tls_client_channel_negotiation_error_expired (Failed)
	 90 - tls_client_channel_negotiation_error_wrong_host (Failed)
	 91 - tls_client_channel_negotiation_error_wrong_host_with_ca_override (Failed)
	 92 - tls_client_channel_negotiation_error_self_signed (Failed)
	 93 - tls_client_channel_negotiation_error_untrusted_root (Failed)
	 94 - tls_client_channel_negotiation_error_untrusted_root_due_to_ca_override (Failed)
	 99 - tls_client_channel_negotiation_error_broken_crypto_rc4 (Failed)
	100 - tls_client_channel_negotiation_error_broken_crypto_rc4_md5 (Failed)
	101 - tls_client_channel_negotiation_error_broken_crypto_dh480 (Failed)
	102 - tls_client_channel_negotiation_error_broken_crypto_dh512 (Failed)
	103 - tls_client_channel_negotiation_error_broken_crypto_null (Failed)
	104 - tls_client_channel_negotiation_error_tls13_to_tls12_server (Failed)
	105 - tls_client_channel_negotiation_error_override_legacy_crypto_tls11 (Failed)
	106 - tls_client_channel_negotiation_error_no_subject (Failed)
	108 - tls_client_channel_negotiation_error_no_common_name (Failed)
	117 - test_duplicate_cert_import (Subprocess aborted)
	118 - tls_channel_echo_and_backpressure_test (Failed)
	119 - tls_channel_shutdown_with_cache_test (Failed)
	120 - tls_channel_shutdown_with_cache_window_update_after_shutdown_test (Failed)
	122 - tls_server_multiple_connections (Failed)
	127 - test_rsa_2048_cert_import (Failed)
	128 - test_rsa_4096_cert_import (Failed)
	129 - test_ecc_p256_cert_import (Failed)
	130 - test_ecc_p384_cert_import (Failed)
	131 - test_ecc_p521_cert_import (Failed)
	133 - test_concurrent_cert_import (Subprocess aborted)
	134 - test_pkcs8_import (Failed)
	135 - test_pkcs12_import (Failed)
	138 - tls_certificate_chain_test (Failed)
	179 - shared_library_open_success (Failed)
	180 - shared_library_find_function_failure (Failed)
	181 - shared_library_find_function_success (Failed)
Errors while running CTest

I.e. most we get identical failures, regardless of my patches being applied. A few extra tests fail on the old system, which may or may not be related to the patches, need to look into that.