Scheduler burst DDos-ing kafka connectors #1710
Description
Issue submitter TODO list
- I've searched for an already existing issues here
- I'm running a supported version of the application which is listed here and the feature is not present there
Is your proposal related to a problem?
I have multiple kafka-connect instances and each of them is loaded with many kafka connector tasks. My ALB is returning 502 errors for many of them. In logs, I can see that KafkaUI is burst sending too many GET requests to get various kafka-connect endpoints. I assume, that my kafka connect instances are busy pipelining data and doesn't have enough resources to reply to all requests in these bursts.
I have traced these requests coming from https://github.com/kafbat/kafka-ui/blob/main/api/src/main/java/io/kafbat/ui/service/ClustersStatisticsScheduler.java. I really like that scheduler is configurable from default 30s here: @Scheduled(fixedRateString = "${kafka.update-metrics-rate-millis:30000}")
Describe the feature you're interested in
It would be great, if I could have another config on the scheduler to set the rate maxRatesPerSecond of these requests per second, so that it doesn't overwhelms my kafka-connectors. Or maybe something like maxConcurrency so that it will get the data in serial.
Alternatively, it would be great if the ClustersStatisticsScheduler would handle these requests more intelligently, like back-off and retries automatically. What is the point of burst sending 40 requests onto Kafka connect instances if they are all responding with 502 errors?
Describe alternatives you've considered
I am setting this in cloudformation Environment,
- Name: KAFKA_UPDATE_METRICS_RATE_MILLIS
Value: "300000" # poll every 5 minutes instead of every 30 seconds
to overwrite default value, but that is not exactly solving the issue, just making it to happen every 5 minutes, instead of every 30 seconds.
Version you're running
FROM kafbat/kafka-ui:2e62ea1
Additional context
No response