Cruise control rebalancing behavior during high disk with some broker

[kspmmitt]

Hello, Can you please confirm the cruise control behavior understanding for below scenario?

Problem Scenario

Broker 101 is using 90% disk

Broker 102 is at 50%

Broker 103 is only at 30%

This creates:

Risk of disk full or performance degradation on Broker 101

Underutilized resources on other brokers

How Cruise Control Fixes It (Disk or CPU Rebalance)
Step 1: Goal Detection
Cruise Control uses goals like:

DiskUsageDistributionGoal

CpuUsageDistributionGoal

These goals define acceptable thresholds for resource usage variance across brokers.

Cruise Control sees that Broker 101 violates the goal.

Step 2: Generate Rebalance Proposal
Cruise Control analyzes:

Which partitions are hosted on Broker 101

Their size and resource usage

Candidate destination brokers that have available disk or CPU

Then it builds a plan to:

Move replicas of large partitions from Broker 101 to less-loaded brokers (e.g., 102 or 103)

Ensure rack-awareness and replication factors are preserved

It does not move all partitions blindly — it optimizes based on size and impact.

Step 3: Execute Rebalance
If the plan is approved (e.g., not a dry run), Cruise Control:

Begins copying the partition data to the new broker(s)

Waits for data to sync (maintains ISR — In-Sync Replicas)

Once safe, removes the replica from the overloaded broker

Kafka continues to serve traffic during this.

Step 4: Resource Usage Becomes Balanced
After the rebalancing:

Disk/CPU usage across brokers is more evenly distributed

No broker is overloaded

Future scaling is smoother

Example Rebalance Command (for Disk Rebalancing)
bash
Copy
Edit
curl -X POST http://:9090/kafkacruisecontrol/rebalance
-d '{
"goals": ["RackAwareGoal", "DiskUsageDistributionGoal"],
"dryrun": false,
"allow_capacity_estimation": true
}'

[kyguy]

Yes @kspmmitt this is generally correct.