|
public void Shutdown(Exception workerException) |
WorkerChannel.Shutdown will loop through all active invocations and terminate them one by one. This is good to avoid orphaned invocations during a worker restart, but it has introduced a different issue. With a high enough active invocations, the process of looping through these invocations can itself take so long new invocations will fail.
This is because we do not start deferring incoming invocations until after the Shutdown method. Taking too long to process the shutdown creates a time window when all new invocations will fail with Did not find any initialized language workers exception.
We need to both (1) close this window of delaying invocations and (2) see if we can process these invocation terminations asynchronously.
azure-functions-host/src/WebJobs.Script.Grpc/Channel/WorkerChannel.cs
Line 1553 in 5a48772
WorkerChannel.Shutdownwill loop through all active invocations and terminate them one by one. This is good to avoid orphaned invocations during a worker restart, but it has introduced a different issue. With a high enough active invocations, the process of looping through these invocations can itself take so long new invocations will fail.This is because we do not start deferring incoming invocations until after the
Shutdownmethod. Taking too long to process the shutdown creates a time window when all new invocations will fail with Did not find any initialized language workers exception.We need to both (1) close this window of delaying invocations and (2) see if we can process these invocation terminations asynchronously.