Long Time_queued_by_rc Duration Even When Resource Group Has Sufficient Resources

[AndreMouche]

Problem Description

The client doesn't cache any tokens if the resource group token is available, so it must acquire a token every time. This may cause long latency when a retry happens. Each retry requires re-acquiring the token through OnRequestWait, even though the resource group has sufficient quota. The delays from token acquisition (lock contention, PD communication latency, request queuing) accumulate during retries, significantly increasing total latency.

Current Behavior: Token Acquisition on Every Request

In the SendRequest

client-go/internal/client/client_interceptor.go

Lines 45 to 88 in 7793c67

	func (r interceptedClient) SendRequest(ctx context.Context, addr string, req *tikvrpc.Request, timeout time.Duration) (resp *tikvrpc.Response, err error) {
	var ruDetails *util.RUDetails
	resourceGroupName, resourceControlInterceptor, reqInfo := getResourceControlInfo(ctx, req)
	if resourceControlInterceptor != nil {
	consumption, penalty, waitDuration, priority, err := resourceControlInterceptor.OnRequestWait(ctx, resourceGroupName, reqInfo)
	if err != nil {
	return nil, err
	}
	req.GetResourceControlContext().Penalty = penalty
	// override request priority with resource group priority if it's not set.
	// Get the priority at tikv side has some performance issue, so we pass it
	// at client side. See: https://github.com/tikv/tikv/issues/15994 for more details.
	if req.GetResourceControlContext().OverridePriority == 0 {
	req.GetResourceControlContext().OverridePriority = uint64(priority)
	}
	if val := ctx.Value(util.RUDetailsCtxKey); val != nil {
	ruDetails = val.(*util.RUDetails)
	ruDetails.Update(consumption, waitDuration)
	}
	}
	if ctxInterceptor := interceptor.GetRPCInterceptorFromCtx(ctx); ctxInterceptor == nil {
	resp, err = r.Client.SendRequest(ctx, addr, req, timeout)
	} else {
	resp, err = ctxInterceptor.Wrap(func(target string, req *tikvrpc.Request) (*tikvrpc.Response, error) {
	return r.Client.SendRequest(ctx, target, req, timeout)
	})(addr, req)
	}
	if resourceControlInterceptor != nil && resp != nil {
	respInfo := resourcecontrol.MakeResponseInfo(resp)
	consumption, waitDuration, err := resourceControlInterceptor.OnResponseWait(ctx, resourceGroupName, reqInfo, respInfo)
	if err != nil {
	return nil, err
	}
	if ruDetails != nil {
	ruDetails.Update(consumption, waitDuration)
	}
	}
	return resp, err
	}

Here

func (r interceptedClient) SendRequest(ctx context.Context, addr string, req *tikvrpc.Request, timeout time.Duration) (resp *tikvrpc.Response, err error) {
    // ...
    if resourceControlInterceptor != nil {
        // OnRequestWait is called on every invocation to acquire a token
        consumption, penalty, waitDuration, priority, err := resourceControlInterceptor.OnRequestWait(ctx, resourceGroupName, reqInfo)
        // ...
    }
    // ...
}

these can cause Latency Accumulation in Retry Scenarios

• Each retry requires waiting for token acquisition (OnRequestWait)
• Even with sufficient tokens, each acquisition may have delays:
  • Lock contention
  • PD communication latency
  • Concurrent request queuing
• These delays accumulate during retries, causing significant total latency increase

[AndreMouche]

In product environment, Even when quota is sufficient, waitDuration can still be high due to:

1. Lock contention: High concurrency causes lock contention in the resource controller
2. PD communication latency: OnRequestWait may need to communicate with PD to acquire tokens
3. Request queuing: Multiple concurrent requests may queue up even with sufficient quota
4. Retry accumulation: Each retry re-invokes OnRequestWait, and delays accumulate across retries
5. No token caching: The client doesn't cache tokens, so every request (including retries) must acquire a new token, even when quota is available