Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add completed / remaining bytes rebalance metrics #2133

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add completed / remaining bytes rebalance metrics #2133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b295216
Update ExecutionTaskTracker.java
HerveRiviere Mar 4, 2024
7dcb94e
Update Sensors.md
HerveRiviere Mar 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions ...control/src/main/java/com/linkedin/kafka/cruisecontrol/executor/ExecutionTaskTracker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@ public class ExecutionTaskTracker {
public static final String METER_INTRA_BROKER_PARTITION_MOVEMENT_RATE = "intra-broker-partition-movement-rate";
public static final String METER_LEADERSHIP_MOVEMENT_RATE = "leadership-movement-rate";
public static final String METER_PARTITION_DATA_MOVEMENT_RATE = "partition-data-movement-rate-MB";
public static final String GAUGE_ONGOING_EXECUTION_TOTAL_DATA_TO_MOVE = "ongoing-execution-inter_broker_data_movement_total_bytes";
public static final String GAUGE_ONGOING_EXECUTION_FINISHED_DATA_TO_MOVE = "ongoing-execution-inter_broker_data_movement_completed_bytes";
public static final String GAUGE_ONGOING_EXECUTION_DATA_MOVEMENT = "ongoing-execution-inter_broker_data_movement_moving_bytes";
public static final String GAUGE_ONGOING_EXECUTION_REMAINING_DATA_TO_MOVE = "ongoing-execution-inter_broker_data_movement_remaining_bytes";

ExecutionTaskTracker(MetricRegistry dropwizardMetricRegistry, Time time) {
List<ExecutionTaskState> states = ExecutionTaskState.cachedValues();
Expand Down Expand Up @@ -111,6 +115,18 @@ private void registerGaugeSensors(MetricRegistry dropwizardMetricRegistry) {
dropwizardMetricRegistry.register(MetricRegistry.name(EXECUTOR_SENSOR, GAUGE_ONGOING_EXECUTION_IN_NON_KAFKA_ASSIGNER_MODE),
(Gauge<Integer>) () -> !_isKafkaAssignerMode
&& !inExecutionTasks(TaskType.cachedValues()).isEmpty() ? 1 : 0);

// expose rebalance metrics
dropwizardMetricRegistry.register(MetricRegistry.name(EXECUTOR_SENSOR, GAUGE_ONGOING_EXECUTION_TOTAL_DATA_TO_MOVE),
(Gauge<Long>) () -> ( finishedInterBrokerDataMovementInMB()
+ inExecutionInterBrokerDataMovementInMB()
+ remainingInterBrokerDataToMoveInMB() ) * 1024 * 1024 );
dropwizardMetricRegistry.register(MetricRegistry.name(EXECUTOR_SENSOR, GAUGE_ONGOING_EXECUTION_FINISHED_DATA_TO_MOVE),
(Gauge<Long>) () -> finishedInterBrokerDataMovementInMB() * 1024 * 1024 );
dropwizardMetricRegistry.register(MetricRegistry.name(EXECUTOR_SENSOR, GAUGE_ONGOING_EXECUTION_DATA_MOVEMENT),
(Gauge<Long>) () -> inExecutionInterBrokerDataMovementInMB() * 1024 * 1024 );
dropwizardMetricRegistry.register(MetricRegistry.name(EXECUTOR_SENSOR, GAUGE_ONGOING_EXECUTION_REMAINING_DATA_TO_MOVE),
(Gauge<Long>) () -> remainingInterBrokerDataToMoveInMB() * 1024 * 1024 );
}

private void registerMeterSensors(MetricRegistry dropwizardMetricRegistry) {
Expand Down
71 changes: 38 additions & 33 deletions docs/wiki/User Guide/Sensors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,39 +13,44 @@ Cruise Control metrics are useful to monitor the state of Cruise Control itself.

### Executor Sensors

| DESCRIPTION | MBEAN NAME |
|--------------------------------------------------------------------|----------------------------------------------------------------------------------------|
| The number of replica action in progress | kafka.cruisecontrol:name=Executor.replica-action-in-progress |
| The number of leadership action in progress | kafka.cruisecontrol:name=Executor.leadership-action-in-progress |
| The number of replica action pending | kafka.cruisecontrol:name=Executor.replica-action-pending |
| The number of leadership action pending | kafka.cruisecontrol:name=Executor.leadership-action-pending |
| The number of replica action aborting | kafka.cruisecontrol:name=Executor.replica-action-aborting |
| The number of leadership action aborting | kafka.cruisecontrol:name=Executor.leadership-action-aborting |
| The number of replica action aborted | kafka.cruisecontrol:name=Executor.replica-action-aborted |
| The number of leadership action aborted | kafka.cruisecontrol:name=Executor.leadership-action-aborted |
| The number of replica action dead | kafka.cruisecontrol:name=Executor.replica-action-dead |
| The number of leadership action dead | kafka.cruisecontrol:name=Executor.leadership-action-dead |
| Has an ongoing execution in kafka_assigner mode | kafka.cruisecontrol:name=Executor.ongoing-execution-kafka_assigner |
| Has an ongoing execution in non-kafka_assigner mode | kafka.cruisecontrol:name=Executor.ongoing-execution-non-kafka_assigner |
| The number of (all) execution stopped | kafka.cruisecontrol:name=Executor.execution-stopped |
| The number of execution stopped by user | kafka.cruisecontrol:name=Executor.execution-stopped-by-user |
| The number of execution started in kafka_assigner mode | kafka.cruisecontrol:name=Executor.execution-started-kafka_assigner |
| The number of execution started in non-kafka_assigner mode | kafka.cruisecontrol:name=Executor.execution-started-non-kafka_assigner |
| Per broker cap on inter-broker partition movements (deprecated) | kafka.cruisecontrol:name=Executor.inter-broker-partition-movements-per-broker-cap |
| Per broker cap on intra-broker partition movements (deprecated) | kafka.cruisecontrol:name=Executor.intra-broker-partition-movements-per-broker-cap |
| Global cap on leadership movements | kafka.cruisecontrol:name=Executor.leadership-movements-global-cap |
| The time taken to exectute proposals involving broker removal | kafka.cruisecontrol:name=Executor.proposal_execution-time-involve-broker-removal |
| The time taken to exectute proposals involving broker demotion | kafka.cruisecontrol:name=Executor.proposal-execution-time-involve-broker-demotion-only |
| The time taken to exectute proposals | kafka.cruisecontrol:name=Executor.proposal-execution-time |
| Max inter-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=inter-broker-partition-movements-max-concurrency |
| Min inter-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=inter-broker-partition-movements-min-concurrency |
| Average inter-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=inter-broker-partition-movements-avg-concurrency |
| Max intra-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=intra-broker-partition-movements-max-concurrency |
| Min intra-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=intra-broker-partition-movements-min-concurrency |
| Average intra-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=intra-broker-partition-movements-avg-concurrency |
| Max leadership movement concurrency of all brokers | kafka.cruisecontrol:name=leadership-movements-max-concurrency |
| Min leadership movement concurrency of all brokers | kafka.cruisecontrol:name=leadership-movements-min-concurrency |
| Average leadership movement concurrency of all brokers | kafka.cruisecontrol:name=leadership-movements-avg-concurrency |
| DESCRIPTION | MBEAN NAME |
|----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|
| The number of replica action in progress | kafka.cruisecontrol:name=Executor.replica-action-in-progress |
| The number of leadership action in progress | kafka.cruisecontrol:name=Executor.leadership-action-in-progress |
| The number of replica action pending | kafka.cruisecontrol:name=Executor.replica-action-pending |
| The number of leadership action pending | kafka.cruisecontrol:name=Executor.leadership-action-pending |
| The number of replica action aborting | kafka.cruisecontrol:name=Executor.replica-action-aborting |
| The number of leadership action aborting | kafka.cruisecontrol:name=Executor.leadership-action-aborting |
| The number of replica action aborted | kafka.cruisecontrol:name=Executor.replica-action-aborted |
| The number of leadership action aborted | kafka.cruisecontrol:name=Executor.leadership-action-aborted |
| The number of replica action dead | kafka.cruisecontrol:name=Executor.replica-action-dead |
| The number of leadership action dead | kafka.cruisecontrol:name=Executor.leadership-action-dead |
| Has an ongoing execution in kafka_assigner mode | kafka.cruisecontrol:name=Executor.ongoing-execution-kafka_assigner |
| Inter-broker total bytes to reassigned for the ongoing execution | kafka.cruisecontrol:name=Executor.ongoing-execution-inter_broker_data_movement_total_bytes |
| Inter-broker reassigned bytes for the ongoing execution | kafka.cruisecontrol:name=Executor.ongoing-execution-inter_broker_data_movement_completed_bytes |
| Inter-broker in-flight reassignement bytes for the ongoing execution | kafka.cruisecontrol:name=Executor.ongoing-execution-inter_broker_data_movement_moving_bytes |
| Inter-broker remaining bytes to reassign for the ongoing execution | kafka.cruisecontrol:name=Executor.ongoing-execution-inter_broker_data_movement_remaining_bytes |
| Average leadership movement concurrency of all brokers | kafka.cruisecontrol:name=ongoing-execution-inter_broker_data_movement_total_bytes |
| The number of (all) execution stopped | kafka.cruisecontrol:name=Executor.execution-stopped |
| The number of execution stopped by user | kafka.cruisecontrol:name=Executor.execution-stopped-by-user |
| The number of execution started in kafka_assigner mode | kafka.cruisecontrol:name=Executor.execution-started-kafka_assigner |
| The number of execution started in non-kafka_assigner mode | kafka.cruisecontrol:name=Executor.execution-started-non-kafka_assigner |
| Per broker cap on inter-broker partition movements (deprecated) | kafka.cruisecontrol:name=Executor.inter-broker-partition-movements-per-broker-cap |
| Per broker cap on intra-broker partition movements (deprecated) | kafka.cruisecontrol:name=Executor.intra-broker-partition-movements-per-broker-cap |
| Global cap on leadership movements | kafka.cruisecontrol:name=Executor.leadership-movements-global-cap |
| The time taken to exectute proposals involving broker removal | kafka.cruisecontrol:name=Executor.proposal_execution-time-involve-broker-removal |
| The time taken to exectute proposals involving broker demotion | kafka.cruisecontrol:name=Executor.proposal-execution-time-involve-broker-demotion-only |
| The time taken to exectute proposals | kafka.cruisecontrol:name=Executor.proposal-execution-time |
| Max inter-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=inter-broker-partition-movements-max-concurrency |
| Min inter-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=inter-broker-partition-movements-min-concurrency |
| Average inter-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=inter-broker-partition-movements-avg-concurrency |
| Max intra-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=intra-broker-partition-movements-max-concurrency |
| Min intra-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=intra-broker-partition-movements-min-concurrency |
| Average intra-broker partition movement concurrency of all brokers | kafka.cruisecontrol:name=intra-broker-partition-movements-avg-concurrency |
| Max leadership movement concurrency of all brokers | kafka.cruisecontrol:name=leadership-movements-max-concurrency |
| Min leadership movement concurrency of all brokers | kafka.cruisecontrol:name=leadership-movements-min-concurrency |
| Average leadership movement concurrency of all brokers | kafka.cruisecontrol:name=leadership-movements-avg-concurrency |



### LoadMonitor Sensors
Expand Down