RedisClient fails to reconnect

[AbhishekS2304]

I have a redis client application (service A) (using lettuce 6.2.1.RELEASE) which connects to redis container and subscribes a channel. When the redis container is up, service 'A' (running as a container) starts and connects with redis. Everything is fine until now.

When the redis container restarts. Service 'A' goes on reconnecting forever but couldn't reconnect. To restore the connection, I always have to restart service 'A' after the redis container is up. Is there a way to reconnect without restarting service 'A'?

RedisURI REDIS_URI= RedisURI.Builder.redis("redis", 6379).withAuthentication("redisUsername", "redisPassword").build()
ClientResources clientResources = DefaultClientResources.create();
RedisClient redisClient = RedisClient.create(clientResources, REDIS_URI); redisClient.setOptions(ClientOptions.builder().autoReconnect(true).pingBeforeActivateConnection(true).build());
redisClient.connectPubSub();

[mp911de]

Sounds pretty much like a DNS caching issue. Please ensure the resolved IP address matches the refreshed/restarted Redis container. You can check this by enabling debug logging.

[AbhishekS2304]

yes @mp911de , the client is trying to reconnect to the old ip of redis container.

[lettuce-eventExecutorLoop-12-1] INFO io.lettuce.core.protocol.ConnectionWatchdog - Reconnecting, last destination was edge-redis.dns.podman/10.89.0.10:6379
[lettuce-eventExecutorLoop-3-1] WARN io.lettuce.core.protocol.ConnectionWatchdog - Cannot reconnect to [edge-redis/:6379]: Connection refused: edge-redis.dns.podman/10.89.0.10:6379

1. I also thought to write custom logic "whenever the connection is lost, instead of trying to reconnect, I want to create a fresh connection". But as the connection lost and reconnection is internal feature of lettuce library. The client program is unaware of this whole thing and I don't know how can I invoke my custom logic?
2. Could you please let me know how can I configure my redis client so that it resolves the ip first then tries to reconnect to the redis.
3. Is there a way to limit the number retries to reconnect to redis container? Lets say, after x retries (to reconnect to redis container), the lettuce library will throw an exception "Could not reconnect to redis". And the client can catch that exception and tries to create a fresh connection.

[AbhishekS2304]

Hi @mp911de - Could you please help me to resolve this discussion?

[laffonso]

Hello @mp911de and @AbhishekS2304, were you able to handle this problem?
If not, I'd like to add some contribution to this discussion since I'm facing the same problem.

I have the same behavior described by Abhishek:

When the redis container restarts. Service 'A' goes on reconnecting forever but couldn't reconnect. To restore the connection, I always have to restart service 'A' after the redis container is up. Is there a way to reconnect without restarting service 'A'?

And with debug logging I also confirmed the lettuce client is trying to reconnect to the old IP address of redis service.

Additional information

• I'm using redis in cluster mode with two operands and a Spring Boot application for the service 'A'
• In my application.yaml I'm using the service hostname and port so the DNS is resolved for the other node and is cached.

spring:
  data:
    redis:
      cluster:
        nodes:
          - <redis_service_name>:6379

It was possible to confirm in debug that the dnscache used by lettuce is io.netty.resolver.dns.DefaultDnsCache and in AddressResolverGroupProvider.java from lettuce, we have a wrapper of netty's DnsAddressResolverGroup.

I was able to customize the DefaultDnsCache maxTtl by creating a @bean with the following:

@Bean
ClientResourcesBuilderCustomizer customizer()
{
    return clientResourcesBuilder -> clientResourcesBuilder.addressResolverGroup(new DnsAddressResolverGroup(
            new DnsNameResolverBuilder().channelType(Transports.datagramChannelClass())
                    .socketChannelType(Transports.socketChannelClass().asSubclass(SocketChannel.class))
                    .cnameCache(new DefaultDnsCnameCache())
                    .resolveCache(new DefaultDnsCache(0, 1, 0))));
}

And this solves the issue because now the maxTtl is 1 second, and after this 1 second, the cached IP from the old redis service is cleared and the DNS is resolved again.

Is there another way of configuring this maxTtl for DefaultDnsCache? What is the recommended approach?
If there is no other way of configuring it, I think It would be better to have this TTL value exposed as a property, what do you think @mp911de ?