Provide an Option to Override TaskList ActivitiesPerSecond #7296
Description
Is your feature request related to a problem? Please describe.
Activity TaskLists support rate limiting the rate at which activities are dispatched to workers. This is a global rate limit across all workers for that TaskList.
The rate limit is specified by the workers, being included in the PollForActivityTaskRequest. The lowest recently specified value is used.
During incident mitigation we may want to rapidly increase/decrease this value. Changing the worker configuration and deploying them may significantly increase remediation time.
Proposed Solution
We should provide an RPC mechanism to manipulate a TaskList, specifically its ActivitiesPerSecond. We should store this value in the TaskList within the database so that it persists. This value should override whatever the workers specify. We should have some mechanism to subsequently remove this option from a TaskList.
We should additionally provide a CLI command that uses this mechanism to easily support manipulating it.
We should additionally expose this information in the Cadence UI, potentially with the option to manipulate it.
Additional context
There is an RPC Method to manipulate the TaskList Partition Configuration, but not a generic one for TaskLists. We should consider whether we can create one that incorporates this other behavior, providing a more standard way to update TaskLists regardless of the properties.