[Q&A]: make the clients persists when sending updates after the server is down

[gonzaq94]

Python version (python3 -V)

3,7

NVFlare version (python3 -m pip list | grep "nvflare")

2.2.2

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.2.2

Operating system

Ubuntu 18.04.3 LTS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I have my server and clients in a VLAN with a private DNS server that it is unavailable from time to time, which makes the server unreachable.
If the server is suddenly unreachable, the training is stopped because the clients cannot send their updates, which is normal. However, if the server comes back to life, the clients do not try to reach it again, and the training is "blocked". When using the "check_status client" command in the admin, the prompt says that the clients are still running, but the clients are neither training nor sending updates to the server. The clients are still reserving GPU memory though, so it is hard to know when this situation arrives.
Ideally, what I would like is the clients to continue trying to reach the server from time to time, so when its reachable again the training can continue. Or, at least, I would prefer the clients to simply abort the training, so I can easily know that there has been an issue.
Is there a way to obtain this behavior with current NvFlare versions ? Is this actually the expected behavior, and I am experiencing a bug ? Or should I submit this as a feature request ?
Thanks in advance!
Gonzalo

[chesterxgchen]

Have you tried 2.3.x release, we have improved quite a bit for the unreliable connection.
When you say the server is unreachable, I assume you mean the network is unreliable, the server is actual running. In the server usually get heart beat to indicate the client is still running. At certain point, it will timeout. this implies the client is dead. The server will decide to stop the job. If you can try 2.3.x or latest 2.4.0 release candidate. Next 2.4.0 release candidate will be release this month

[gonzaq94]

Thank you for you answer, I'll try 2.3 and/or 2.4.0 and let you know the result. Yes that is correct, the server and clients are always running, but the connection is unreliable.
I have also observed the same issue in other direction: the server cannot reach the clients for a while (say 8 hours), and when it does reach them, one of the clients is down. I guess this is because of an expired timeout.
Are these issues solved in 2.3 or 2.4.0 ? Is there a way to increase this timeout ?

[chesterxgchen]

both 2.3 and 2.4 has configurable varaibles to change different timeouts. One of customers has the unstable connection issue, the 2.3.x patches addressed the issues. Not sure its the same issue as your. All patches have fold into the 2.4.0. The configurations documentation is to be added to 2.4.0 documentation (working-progress) ( not in 2.3.x documentation). The documentation will be merged before mid-Dec.

[gonzaq94]

Hi, I'm trying the 2.3.8 version. Could you tell be how to adjust the timeouts with the variables that you mentioned ?

[chesterxgchen]

For 2.3.x, we did not document this as the feature was hidden (even it was there), we did not public announce it.
For 2.4.0 we are working on the documentation, the PR is under review. most of the timeout configs are the same as 2.3.x, you can take a look at this PR's communication configuration section: https://github.com/NVIDIA/NVFlare/pull/2194/files#diff-207df75e42daf833ddfbc9129556853b5f72d5d5cc701931e91f993042fa52ba

The PR will be merged in next few days

[gonzaq94]

Hi, I have been having a look at the config documentation you sent. I have been able to change some settings in the config_fed_client.json (e.g., runner_sync_timeout, max_runner_sync_tries, etc.). However, some other settings need to be set/changed in <site_workspace>/local/comm_config.json, <site_workspace>/local/resources.json, but those .jsons are not present in the the indicated folder (<site_workspace>/local/), which has been generated automatically when using nvflare provision.

In the case of the resources.json, there is a file called <site_workspace>/local/resources.json.default. Would it be the same if I edit this file ? In the case of the comm_config.json, there no similar file in the directory. Should I create a new file called comm_config.json and edit it ?

Btw, I am using nvflare 2.3.8. Is it possible that these other options are only available in 2.4.0 ?

Thanks again,

Gonzalo

[chesterxgchen]

Thanks Sean, can you add code example to the doc ? the message-parameters are just segment, it unclear where to put in the overall comm_config.json file.

[gonzaq94]

Thanks for your answers. I'll let you know if the connection stability improves after these changes. After reading you answers and the documentation, I have two additional questions:

1. Shouldn't I also include a resources.json in the server ?
2. Shouldn't I modify the "hearbeat_interval" parameter ?
   Thanks.
   Gonzalo

[chesterxgchen]

You can certainly add resource.json in the server if you like to custom configuration instead of using the default. in many cases, server doesn't do the heavy lifting (as client machine does the training), so there is no need to make changes to the resource.json

"hearbeat_interval" parameter value may need to adjust based on your network.

[gonzaq94]

An update from my side: these changes effectively improved the connection stability between my clients and server. Thank you for your answers!