log: add logging to failover module

[vakhov]

What has been done:
Added structured logging to failover.lua and coordinator.lua. Now logs cover leader switches (with aliases and URIs), quorum and health checks, immunity status, and errors during failover decisions.

Why:
Previously, the failover process lacked visibility. It was hard to trace leader election logic or investigate why a leader was changed without deep manual inspection.

What problem is being solved:
This change improves observability of failover and leader appointment workflows. It makes it easier to debug production incidents and analyze unexpected leader transitions.

I didn't forget about

• Tests
• Changelog
• Documentation

Close TNTP-3341

[Satbek]

There is also roles/coordinator.lua where locates coordinator's logic.
I think we should add the reason for appointment in log message

https://github.com/tarantool/cartridge/blob/763de81461d48ac5d6aa219ca52f0f4ff8f28a6b/cartridge/roles/coordinator.lua#L106C18-L109C18

something like:

previous leader instance storage-b2(addr1.buiseness.net:3301) abcd-abcd-1234-1234 id dead, appoint new leader ...

and add info about previous leader for manual leader change

cartridge/cartridge/roles/coordinator.lua

Lines 381 to 384 in 763de81

	log.info('Replicaset %s: appoint %s (%q) (manual)',
	replicaset_uuid, decision.leader,
	assert(servers[decision.leader]).uri
	)

[Satbek]

Thanks for logging improvements.

I think we should log here an instance alias too

https://github.com/tarantool/cartridge/pull/2339/files#diff-b3b13e336d0031b78b83b2caa6fff8da5ce604376fecbe1c83723d0ab263a522L269-L274

[Satbek]

Maybe we should also log more details about make_decision—for instance, how many candidates were examined, in case the whole replica set looks dead and that’s why no decision was made, or because immunity_timeout hasn’t expired yet.

We should also log that the control_loop has started and finished; this occurs when membership notifications arrive.

Could you also attach an example of what the logs will look like in the merge request?

[vakhov]

Maybe we should also log more details about make_decision—for instance, how many candidates were examined, in case the whole replica set looks dead and that’s why no decision was made, or because immunity_timeout hasn’t expired yet.

We should also log that the control_loop has started and finished; this occurs when membership notifications arrive.

Could you also attach an example of what the logs will look like in the merge request?

I’ve added the requested improvements:

• Logs in make_decision now include:

  • how many candidates were evaluated,
  • whether immunity is still active,
  • a warning if no healthy candidates were found.

• control_loop logs when it starts and finishes each iteration.

Example of how the logs look now:

control_loop: started
make_decision: evaluating replicaset bbbbbbbb-0000-0000-0000-000000000000 (candidates=3)
make_decision: immunity not expired for leader bbbbbbbb-bbbb-0000-0000-000000000001 (storage-1, localhost:13302) (expires in 1.0 sec)
control_loop: finished

control_loop: started
make_decision: evaluating replicaset bbbbbbbb-0000-0000-0000-000000000000 (candidates=3)
make_decision: no healthy candidates found in replicaset bbbbbbbb-0000-0000-0000-000000000000
control_loop: finished

control_loop: started
make_decision: evaluating replicaset bbbbbbbb-0000-0000-0000-000000000000 (candidates=3)
control_loop: replicaset bbbbbbbb-0000-0000-0000-000000000000: appoint new leader bbbbbbbb-bbbb-0000-0000-000000000001 (storage-1, localhost:13302), previous leader bbbbbbbb-bbbb-0000-0000-000000000002 (storage-2, localhost:13303)
control_loop: finished

Let me know if you’d like any further adjustments!

[Satbek]

We also need to log replicaset's alias. It is in topology_cfg so we can add it.

About coordinator:

We need to log why an appointment was made and why it wasn’t.

Log levels

• info — when an appointment is made
• warn — when an appointment is not made

When an appointment is made (info):

• The current leader is unhealthy; after examining N candidates a new one was chosen.
• This is the first appointment; the first node in the topology was selected.

When an appointment is not made (warn):

• The replica set’s immunity timeout hasn’t expired.
• The current leader is alive and electable (may be too frequent to log; absence of such logs can itself imply everything is fine).

Suggested flow:

1. First check if the current leader is healthy.

   • If not, log the problem and proceed.
   • If yes, log that fact and exit.

2. If the leader is unhealthy but still under an immunity timeout, log the timeout and the decision not to switch.

Logs must make it clear:

• How candidates are evaluated
• Which replica sets were considered
• Why the master was changed or kept unchanged.

[Satbek]

Hi,

local control_fiber = fiber.new(control_loop, session)
control_fiber:name('failover-coordinate')

Since the fiber is already named, that name will appear in the logs, so an extra prefix isn’t necessary.

The same applies to fencing_healthcheck:

vars.fencing_fiber = fiber.new(fencing_watch)
vars.fencing_fiber:name('cartridge.fencing')

The fiber name itself makes it clear where the log entry came from.

Let’s also keep the log line format:

Replicaset %s: appoint %s (%q) (manual)

so it always begins with “Replicaset …”.
For manual appointments, it would be helpful to include the replicaset alias as well.

Thanks!

[vakhov]

@Satbek I’ve updated the code as discussed:

• Removed redundant log prefixes like control_loop: (and other) since the fiber names are already visible in log entries.
• Adjusted log messages to always start with Replicaset ... for consistency.
• Cleaned up log formatting to avoid duplication and keep entries concise.

Let me know if you’d like any further tweaks or adjustments!

[vakhov]

Here is an example of the log output with the current changes applied:

Making decisions
Replicaset 11111111-2222-3333-4444-555555555555(storage-replicaset): no appointment made (reason=immunity_not_expired, checked=0)
Wait membership notifications

Making decisions
Replicaset 11111111-2222-3333-4444-555555555555(storage-replicaset): no appointment made (reason=no_healthy_candidates, checked=3)
Wait membership notifications

Making decisions
Replicaset 11111111-2222-3333-4444-555555555555(storage-replicaset): appoint new leader bbbbbbbb-bbbb-cccc-dddd-eeeeeeeeeeee (storage-2, "localhost:3302"), previous leader aaaaaaaa-aaaa-bbbb-cccc-dddddddddddd (storage-1, "localhost:3301") (reason=new_leader_selected, checked=2)
Wait membership notifications

[Satbek]

LGTM

please, update changelog