[Q&A] Client continues attempts to connect the server forever, even when the server is really unreachable.

[evgvain]

Python version (python3 -V)

3.12

NVFlare version (python3 -m pip list | grep "nvflare")

2.5.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 22.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hi! I am running Flare in distributed (production) mode. My system launches new client process for every federated computation and shouts them down when the computation is done.
In some cases, the server is unreachable due to missing connectivity (external network issues). The client process logs the exception as expected:

File "xxxx/lib/python3.12/site-packages/nvflare/private/fed/client/fed_client_base.py", line 167, in set_sp
    self._create_cell(location, scheme)
  File "xxxx/lib/python3.12/site-packages/nvflare/private/fed/client/fed_client_base.py", line 230, in _create_cell
    raise RuntimeError(f"Failed to get engine after {time.time() - start} seconds")
RuntimeError: Failed to get engine after 15.004836797714233 seconds

However it continues attempts to connect forever:

2024-12-24 10:57:24,935 - CoreCell - WARNING - [ME=site-1 O=? D=server F=? T=? CH=task TP=challenge SCH=? STP=? SEQ=?] no connection to server
2024-12-24 10:57:24,935 - CoreCell - ERROR - [ME=site-1 O=site-1 D=server F=site-1 T=server CH=task TP=challenge SCH=? STP=? SEQ=?] cannot send to 'server': target_unreachable
2024-12-24 10:57:24,936 - Communicator - INFO - challenge result: target_unreachable 
2024-12-24 10:57:24,936 - Communicator - INFO - re-challenge after 2.0 seconds

How do I cause to client stop in this case and in general on any run-time error?
Thanks.

[chesterxgchen]

If the network is gone, and client is designed to continue to re-connect, as there is no way for the client to know this is temporary network interrupt or permanent loss.

If you not like to see these messages, should you just simply kill the client process if you don't want them to continue ?