Health watcher primary unreachable

[lynshi]

The healthy_watcher.py script currently returns HealthState.unavailable if the current primary is unreachable. Why is this determined to be an unrecoverable state instead of returning HealthState.election - wouldn't one of the nodes eventually start an election if it never hears from the primary whether or not the primary exists?

[jumaffre]

@lynshi: HealthState.unavailable does not indicate that the service is unrecoverable; it simply states that the current primary is not available and that an observer cannot tell for sure whether an election is in progress at this point in time.

In fact, the get_network_health() functions doesn't decide if the service is unrecoverable but simply returns the current health state of the network. It is up to the run() loop to decide whether the service is unrecoverable, which is declared as such if the service hasn't be HealthState.stable for more than unstable_threshold_s (which is N times the election timeout, with N to be decided by the operator based on how sensitive the health check should be).