Extension - Document High CPU Usage by Tokio Runtime with Default ThreadPool

[huntharo]

Motivations

• This high CPU usage observed with the default runtime is likely a problem others are encountering and may not be aware of
• There likely needs to be an improvement made to the Tokio work stealing algorithm to reduce stealing when there is not enough work to go around to all the threads

Video Review of High Rust Tokio Async Runtime CPU - YouTube

Click to view on YouTube

Links

• Multithreaded runtime cause high CPU usage tokio-rs/tokio#4569
• Avoiding async entirely rust-lang/wg-async#58
• https://www.csd.uwo.ca/~mmorenom/CS433-CS9624/Resources/Scheduling_multithreaded_computations_by_work_stealing.pdf
• https://www.cs.cmu.edu/~guyb/papers/locality2000.pdf
• https://people.csail.mit.edu/shanir/publications/Work_Queues.pdf

Test Setup

• 10 max concurrent requests from Router to Extension
  • DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=5 LAMBDA_DISPATCH_ChannelCount=10 LAMBDA_DISPATCH_MaxConcurrentCount=10 LAMBDA_DISPATCH_AllowInsecureControlChannel=true LAMBDA_DISPATCH_PreferredControlChannelScheme=http LAMBDA_DISPATCH_FunctionName=dogs AWS_LAMBDA_SERVICE_URL=http://localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Router/bin/Release/net8.0/PwrDrvr.LambdaDispatch.Router 2>&1 | tee router.log
• 10 concurrent connections from oha
  • oha -c 10 -z 60s http://127.0.0.1:5001/ping

Test Cases

Rust Extension with Default Tokio Threading

• 14,000 RPS
• ❌ 270% CPU
• Command line: LAMBDA_DISPATCH_RUNTIME=default_multi_thread LAMBDA_DISPATCH_FORCE_DEADLINE=60 AWS_LAMBDA_FUNCTION_VERSION=\$LATEST AWS_LAMBDA_FUNCTION_MEMORY_SIZE=512 AWS_LAMBDA_FUNCTION_NAME=dogs AWS_LAMBDA_RUNTIME_API=localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token cargo run --release --bin extension 2>&1 | tee extension.log

Rust Extension with Current Thread Tokio Threading

• 19,000 RPS
• ✅ 90% CPU
• Command line: LAMBDA_DISPATCH_RUNTIME=current_thread LAMBDA_DISPATCH_FORCE_DEADLINE=60 AWS_LAMBDA_FUNCTION_VERSION=\$LATEST AWS_LAMBDA_FUNCTION_MEMORY_SIZE=512 AWS_LAMBDA_FUNCTION_NAME=dogs AWS_LAMBDA_RUNTIME_API=localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token cargo run --release --bin extension 2>&1 | tee extension.log

DotNet Extension

• 14,000 RPS
• ❌ 310% CPU
• Command line: DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0 AWS_LAMBDA_RUNTIME_API=localhost:5051 AWS_REGION=us-east-2 AWS_ACCESS_KEY_ID=test-access-key-id AWS_SECRET_ACCESS_KEY=test-secret-access-key AWS_SESSION_TOKEN=test-session-token src/PwrDrvr.LambdaDispatch.Extension/bin/Release/net8.0/bootstrap 2>&1 | tee extension.log