Bug: Ruler showing incredible amount of network traffic

[ecktom]

What is the bug?

We've recently deployed beyla into our observability cluster running Mimir to gather some metrics about how traffic flows. A really great tool! Kudos to the team!

This brought up some interesting results. Ruler is by far our top consumer in terms of data coming in. Also the index-cache has much more traffic flowing than I would have expected but this is maybe a question for some other day ;)

Looking at the ruler specifically we can see that most traffic is coming in from the store-gateways followed by the ingesters.

We than figured out that we've configured compression a long time back between distributor and ingesters using -ingester.client.grpc-compression=snappy on the distributor but apparently never did so for the ruler as we were not using it at that time.
So we moved ahead and added -ruler.client.grpc-compression=snappy and -ruler.query-frontend.grpc-client-config.grpc-compression=snappy at the ruler which surprisingly had 0 effect.
Next thing we tried was adding:

ruler:
  ruler_client:
    grpc_compression: snappy
  query_frontend:
    address: "dns:///gridx-sre-mimir-query-frontend.mimir:9095"
    grpc_client_config:
      grpc_compression: snappy

to the structured config and restarting the ruler which did the trick. Looking at CPU/Memory/Network metrics of the ruler we saw some unexpected results, all of them improved massively (we applied the change at 16:45 and rolled back at 17:33). The change of CPU/Memory is a mystery for me.

However, it seems we've just moved the issue somewhere else... ;) At the same time we noticed that the queriers are receiving much more traffic in a very similar proportion.

Much more traffic is flowing between queriers and ingester/store-gateways which we can also see on the Mimir Reads Dashboard in terms of RPS of all of those components on the read path.

Looking at the Mimir Ruler dashboard it looks like the ruler just stopped doing requests

Total traffic is much worse than before as also traffic between the store-gateway and the index-cache increased a lot

We tried adding frontend.grpc_client_config.grpc_compression and frontend_worker.grpc_client_config.grpc_compression on top which did not seem to have any effect after restarting the queriers and the query-frontends. Is there anything we're missing or which could explain this pattern?

What was your environment?

• AWS
• Kubernetes
• mimir-distributed-5.5.1
• 2.14.2

Any additional context to share?

No response

[deniszh]

In my experience grpc client config in Mimir is quite cryptic. I would recommend to pick up one option (either config or cli) and don't mix it up without need. Try to configure snappy compression everywhere in config, it's lightweight enough to not bring too much load and recommended for production.
Something like

    alertmanager:
      alertmanager_client:
        grpc_compression: snappy
    frontend:
      grpc_client_config:
        grpc_compression: snappy
    frontend_worker:
      query_scheduler_grpc_client_config:
        grpc_compression: snappy
      grpc_client_config:
        grpc_compression: snappy
    ingester_client:
      grpc_client_config:
        grpc_compression: snappy
    ruler:
      query_frontend:
        grpc_client_config:
          grpc_compression: snappy
      ruler_client:
        grpc_client_config:
          grpc_compression: snappy
    query_scheduler:
      grpc_client_config:
        grpc_compression: snappy

(I tried to make config proper but please check it before going full prod).
This config should work for both external and internal ruler modes.

[ecktom]

Thanks for your response @deniszh, highly appreciated! I think we understood now what was going on here:

• -ruler.query-frontend.grpc-client-config.grpc-compression=snappy did not have any effect at we missed to configure the query_frontend address which is needed to implicitly turn on the external mode. Thus the config had no effect. I however agree, it's quiet cryptic ;)
• We figured out that we had some recording rules were bad and looked at way to much data. Thus the amount of data traffic we saw on the ruler was quiet real. Fixing those recording rules dropped the data traffic.
• AFAIK there is not much we can do in terms of compression on the read path (this is something you could maybe confirm). From what I understood, on the read path the chunks are in TSDB format are already quiet compressed, thus adding eg. snappy for communication between the querier and the store-gateway/ingester would not do much?

[deniszh]

AFAIK there is not much we can do in terms of compression on the read path (this is something you could maybe confirm)

Sounds logical, but can't confirm or deny.