bug: AWS_ERROR_HTTP_STREAM_MANAGER_CONNECTION_ACQUIRE_FAILURE occurring regularly

[dayaffe]

Describe the bug

Swift SDK supports 2 HTTP clients -- URLSession and CRT's HTTP Client. One of our integration tests uses AWS Transcribe and attempts to concurrently send 25 streaming requests. This test never fails on URLSession but fails frequently (not always) on CRT's http client. After some initial debugging it seems the issue arises because the stream manager tries to acquire a connection after the stream is already closed. We've already attempted to fix this from the SDK side with no luck so we are hoping you might have some more insight. One thing we've done is add error code 2087 to the list of CRT errors we retry which has not lowered the frequency of the issue.

Error:
<EXPR>:0: error: TranscribeStreamingTests.test_25xConcurrent_streamTranscription : threw error "crtError(AwsCommonRuntimeKit.CRTError(code: 2087, message: "Stream acquisition failed because stream manager failed to acquire a connection", name: "AWS_ERROR_HTTP_STREAM_MANAGER_CONNECTION_ACQUIRE_FAILURE"))"

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Stream manager should not be failing to acquire connection so often.

Current Behavior

Stream manager often fails to acquire a connection.

Reproduction Steps

https://github.com/awslabs/aws-sdk-swift/blob/9ad12684f6cb9c9b60e840c051a2bba604024650/IntegrationTests/Services/AWSTranscribeStreamingIntegrationTests/TranscribeStreamingTests.swift#L26

Possible Solution

No response

Additional Information/Context

0_linux (jammy, 5.9).txt.zip
Logs with trace

aws-crt-swift version used

latest

Compiler and Version used

latest

Operating System and version

linux