Optimization proposal readiness

[kspmmitt]

Hello,

1. Is it safe and ok to run rebalance if optimization proposal is ready even if there are errors as below or some other errors in cruise control POD?
   OR optimization proposal will not be created in these cases or errors?

2. Also, below errors are not talked about in troubleshooting guide. ()

3. Sometimes kafka rebalance optimization status shows no data to move (0 MB) but only replicas count. Is this fine? As moving replicas means data contained by them will also move?

4. In one test scenario, there were 3 brokers initially, one more broker added and after rebalancing it is noticed that "Broker 3 experienced an increase in partition load, while others (like Broker 0, Broker 1, and Broker 2) lost leader and replica partitions". See the attached full brokerLoad.json and the analysis given by Gen AI and further verified manually myself.
   rebalanced.docx

{"version": "1.2.0", "timestamp": "2025-08-02T02:58:49.079+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "Failed to produce partition metric sample for __consumer_offsets-22 of timestamp 1754103528270 due to exception org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.", "metadata": {"container_name": "cruise-control", "pod_name": "eric-bss-em-kb-operand-emoperand-cruise-control-5db5d775b-8sgr6"}, "extra_data": {"file": "AbstractKafkaSampleStore.java", "line": "117", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

{"version": "1.2.0", "timestamp": "2025-08-02T02:58:49.081+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "Failed to produce partition metric sample for strimzi.cruisecontrol.modeltrainingsamples-16 of timestamp 1754103528270 due to exception org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.", "metadata": {"container_name": "cruise-control", "pod_name": "eric-bss-em-kb-operand-emoperand-cruise-control-5db5d775b-8sgr6"}, "extra_data": {"file": "AbstractKafkaSampleStore.java", "line": "117", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

{"version": "1.2.0", "timestamp": "2025-08-02T02:58:49.080+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "Failed to produce partition metric sample for __transaction_state-45 of timestamp 1754103528270 due to exception org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.", "metadata": {"container_name": "cruise-control", "pod_name": "eric-bss-em-kb-operand-emoperand-cruise-control-5db5d775b-8sgr6"}, "extra_data": {"file": "AbstractKafkaSampleStore.java", "line": "117", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

[kyguy]

Is it safe and ok to run rebalance if optimization proposal is ready even if there are errors as below or some other errors in cruise control POD?
OR optimization proposal will not be created in these cases or errors?

An optimization proposal will only be created if it satisfies the goal configurations in your rebalance request or Cruise Control server configuration.

Also, below errors are not talked about in troubleshooting guide. ()
...
{"version": "1.2.0", "timestamp": "2025-08-02T02:58:49.079+00:00", "severity": "error", "service_id": "eric-data-messagebus-kf", "message": "Failed to produce partition metric sample for __consumer_offsets-22 of timestamp 1754103528270 due to exception org.apache.kafka.common.errors.NotEnoughReplicasException: Messages are rejected since there are fewer in-sync replicas than required.", "metadata": {"container_name": "cruise-control", "pod_name": "eric-bss-em-kb-operand-emoperand-cruise-control-5db5d775b-8sgr6"}, "extra_data": {"file": "AbstractKafkaSampleStore.java", "line": "117", "mbkf_cluster_name": "eric-bss-em-kb-operand-emoperand"}}

This is more of an issue related to the Kafka cluster replication than Cruise Control itself. Either the cluster replication is lagging or the the min.insync.replicas configuration is too high given the number of configured replicas. If you attach your full broker and Cruise Control server configurations I can take a close look if you would like.

In one test scenario, there were 3 brokers initially, one more broker added and after rebalancing it is noticed that "Broker 3 experienced an increase in partition load, while others (like Broker 0, Broker 1, and Broker 2) lost leader and replica partitions". See the attached full brokerLoad.json and the analysis given by Gen AI and further verified manually myself.

It sounds like some of the partitions/load was migrated to "Broker 3" as part of the partition rebalance. This is intended behavior.

[kspmmitt]

Yes, I am using strimzi operator-based Kafka and in Kubernetes type deployment.

Regarding my query 3 and 4 above -
3 > replica moves meaning data will also move but still it shows data moved as 0 MB, is this right understanding?

4 > As I see in brokerLoad.json, it seems that new added broker got overloaded now w.r.t leaders assignment but it was supposed to be even distribution or?

[kyguy]

3 > replica moves meaning data will also move but still it shows data moved as 0 MB, is this right understanding?

From what I understand, the value of that field is rounded to the closest integer. If there is less than a megabyte of data being moved as part of the rebalance, the value of that field will be 0.

4 > As I see in brokerLoad.json, it seems that new added broker got overloaded now w.r.t leaders assignment but it was supposed to be even distribution or?

It depends on the goals that were configured in the Cruise Control server configurations or rebalance request. Partition leadership is only one variable or many that are taken into account when Cruise Control decides where to move what partitions where. Another important consideration is whether the Cruise Control resource capacity settings accurately reflect the actual capacity of the cluster resources. A misconfiguration here could lead Cruise Control to make poor partition movement suggestions.