CPU-bound tasks seem to starve network-bound tasks

[Methuselah96]

Investigative information

Please provide the following:

• Timestamp: 12/12/20 2:56:49 PM ET
• Function App version: 3
• Function App name: NpgsqlHeavyCPU
• Function name(s) (as appropriate): NpgsqlOrchestration
• Invocation ID: 9948a45e-dcf9-410f-a167-ea81f2dc4682
• Region: East US 2

Repro steps

I created a sample project to demonstrate the problem: https://github.com/Methuselah96/NpgsqlHeavyCpu

The sample only shows Npgsql, but I've also seen issues with Cosmos and Blob Storage as well.

Expected behavior

The Azure Function to complete without failure.

Actual behavior

Various socket-related errors:
Npgsql:

Exception message: Npgsql.NpgsqlException (0x80004005): Exception while writing to stream
Stack trace:
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)

at Npgsql.NpgsqlWriteBuffer.Flush(Boolean async)

at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnection.c__DisplayClass32_0.g__OpenLong|0>d.MoveNext()

--- End of stack trace from previous location where exception was thrown ---

at Dapper.SqlMapper.QueryRowAsync[T](IDbConnection cnn, Row row, Type effectiveType, CommandDefinition command) in /_/Dapper/SqlMapper.Async.cs:line 482

Npgsql:

Exception message: System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
Stack trace:
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at System.Net.FixedSizeReader.ReadPacketAsync(Stream transport, AsyncProtocolRequest request)

at System.Net.Security.SslStream.ThrowIfExceptional()

at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)

at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result)

at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)

at System.Net.Security.SslStream.c.b__64_2(IAsyncResult iar)

at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)

--- End of stack trace from previous location where exception was thrown ---

at Npgsql.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.ConnectorPool.AllocateLong(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)

at Npgsql.NpgsqlConnection.c__DisplayClass32_0.g__OpenLong|0>d.MoveNext()

Cosmos:

System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.DocumentClient.HttpRequestMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Microsoft.Azure.Cosmos.GatewayAccountReader.GetDatabaseAccountAsync(Uri serviceEndpoint)

at Microsoft.Azure.Cosmos.Routing.GlobalEndpointManager.GetDatabaseAccountFromAnyLocationsAsync(Uri defaultEndpoint, IList`1 locations, Func`2 getDatabaseAccountFn)

at Microsoft.Azure.Cosmos.GatewayAccountReader.InitializeReaderAsync()

at Microsoft.Azure.Cosmos.CosmosAccountServiceConfiguration.InitializeAsync()

at Microsoft.Azure.Cosmos.DocumentClient.InitializeGatewayConfigurationReaderAsync()

at Microsoft.Azure.Cosmos.DocumentClient.GetInitializationTaskAsync(IStoreClientFactory storeClientFactory)

at Microsoft.Azure.Cosmos.DocumentClient.EnsureValidClientAsync()

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.EnsureValidClientAsync(RequestMessage request)

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler.SendAsync(Uri resourceUri, ResourceType resourceType, OperationType operationType, RequestOptions requestOptions, ContainerCore cosmosContainerCore, Nullable`1 partitionKey, Stream streamPayload, Action`1 requestEnricher, CosmosDiagnosticsContext diagnosticsContext, CancellationToken cancellationToken)

at Microsoft.Azure.Cosmos.CosmosClient.c__DisplayClass27_0.b__0>d.MoveNext()

Blob storage:

Microsoft.WindowsAzure.Storage.StorageException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Http.HttpRequestException: An attempt was made to access a socket in a way forbidden by its access permissions.

---> System.Net.Sockets.SocketException (10013): An attempt was made to access a socket in a way forbidden by its access permissions.

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

--- End of inner exception stack trace ---

at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)

Known workarounds

For Npgsql, if I opened all the connections before running the CPU-bound code, then I wouldn't have any errors. I haven't found any workarounds for Cosmos or Blob storage yet.

[v-bbalaiagar]

Hi @Methuselah96 , Apologies for the delayed response, the issue was somehow lost in the trace. Can you tell us if you are still facing this issue.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[Methuselah96]

I'm working on testing this again to see if the issue still exists.

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[Methuselah96]

@v-bbalaiagar I have verified that we are still having the issue. I have updated the sample project with the latest version of each package, an upgrade to Azure Functions 4, and a more helpful README.