How cruise control handles a solved event

[mbribeiro]

We are trying to integrate cruise control alert reports with a custom python script we have made. We are using alerta.io integration for reporting problems to this custom script (basically a python webhook). We were able to receive open error events but these events remain forever opened and are never reported as solved. My question is: how cruise control handles to fix alerts (may be for alerta.io point of view)? Does it send a POST request event with the same ID and a different severity ? Or it sends a DELETE request with the same ID ? We are asking this because we did generate a critical alert, fixed the problem and put our webhook in debug mode and nothing regarding a close event was received.

[kyguy]

We were able to receive open error events but these events remain forever opened and are never reported as solved.

From what I understand, the Cruise Control anomaly notifier - including the AlertaSelfHealingNotifier class [1] - currently supports sending open alerts when an issue is detected, but does not send any follow-up messages when those alerts have been addressed or resolved.

To close out those events in Alerta from the Cruise Control side, we would need to enhance the AlertaSelfHealingNotifier to detect that the anomaly has been addressed (or doesn't exist anymore) and send a close message to Alerta.

[1] https://github.com/linkedin/cruise-control/blob/main/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/detector/notifier/AlertaSelfHealingNotifier.java