-
Notifications
You must be signed in to change notification settings - Fork 78
Description
Currently, the default deadline for gRPC client stubs is set at stub creation time via DefaultDeadlineSetupClientInterceptor, and the Duration value
is captured in the interceptor instance. This means that changing the deadline configuration at runtime (e.g., via JMX, Spring Actuator, or dynamic
configuration) requires recreating all affected stubs to pick up the new value.
This limitation creates operational challenges in production environments where adjusting timeouts without service interruption is often necessary.
Current Behavior
- Application starts and creates stubs with
spring.grpc.client.channels.my-service.default-deadline=5s - Stub is created with a
DefaultDeadlineSetupClientInterceptorthat capturesDuration.ofSeconds(5) - At runtime, configuration is changed to
default-deadline=10s - Existing stubs continue using 5s deadline because the
Durationis already baked into the interceptor - Only newly created stubs will use the 10s deadline
Code Example
// Application startup
@GrpcClient("my-service")
private MyServiceBlockingStub stub; // Created with 5s deadline
// Runtime: Change deadline via configuration
// spring.grpc.client.channels.my-service.default-deadline=10s
// Problem: stub still uses 5s deadline
stub.myMethod(request); // ❌ Still uses 5s, not 10sExpected Behavior
When the deadline configuration changes at runtime, all RPC calls (including those from existing stub instances) should automatically use the new
deadline value without requiring:
- Stub recreation
- Application restart
- Traffic interruption
Use Cases
- Production tuning: Adjusting timeouts based on observed latencies without restarting services
- Incident response: Quickly increasing deadlines during degraded downstream service conditions
- A/B testing: Experimenting with different timeout values in production
- JMX/Actuator integration: Runtime management via operational tools
Proposed Solution
Modify DefaultDeadlineSetupClientInterceptor to read the deadline from GrpcClientProperties on each call, rather than capturing it at construction
time.
Suggested Implementation
public class DefaultDeadlineSetupClientInterceptor implements ClientInterceptor {
private final GrpcClientProperties properties;
private final String channelName;
public DefaultDeadlineSetupClientInterceptor(
GrpcClientProperties properties,
String channelName) {
this.properties = properties;
this.channelName = channelName;
}
@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions,
Channel next) {
// Read deadline fresh on each call
GrpcClientProperties.ChannelConfig config =
properties.getChannels().get(channelName);
if (config != null && config.getDefaultDeadline() != null) {
Duration deadline = config.getDefaultDeadline();
if (callOptions.getDeadline() == null) {
callOptions = callOptions.withDeadlineAfter(
deadline.toMillis(),
TimeUnit.MILLISECONDS
);
}
}
return next.newCall(method, callOptions);
}
}Alternative: Configuration Flag
If backward compatibility is a concern, this could be enabled via configuration:
spring.grpc.client.dynamic-deadline=trueWhen enabled, the interceptor reads from config on each call. When disabled (default), it uses the current behavior.
Workarounds (Current)
We currently work around this limitation by:
-
Recreating stubs: Remove cached stubs and recreate with new deadline
⚠️ Causes brief traffic interruption⚠️ Complex synchronization issues⚠️ Only works if application doesn't hold stub references
-
Restart application: Redeploy with new configuration
⚠️ Significant downtime⚠️ Not practical for production tuning
Environment
- Spring gRPC version: 0.12.0
- Spring Boot version: 3.x
- gRPC Java version: 1.x