[Q&A] Admin command remove_client during job running

[jingyunc]

Python version (python3 -V)

3.8

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.0rc3

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 20.04

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

In real-world mode, while running the Brats18 example (with 4 sites on a single workstation), I ran remove_client site-4 on admin console. The server side showed following messages:

ClientManager - INFO - Client Name:site-4 	Token: 6a5e9a78-986e-4184-bc1a-63c933b3f1ca left.  Total clients: 3
ClientManager - INFO - Re-activate the client:site-4 with token: 6a5e9a78-986e-4184-bc1a-63c933b3f1ca

However, site-4 didn't get removed from the job, instead it continued to train and send model update, until the whole job was completed.

My main goal was to test site drop-out in real-world amid running job, is remove_client the right command for that? Thanks.

[chesterxgchen]

@yhwen can you check this ?

[YuanTingHsieh]

@jingyunc thanks for the question.

Short answer to "is remove_client the right command for that?": No

What remove_client did is just remove the reference from the server side, it is ONLY useful when the client is dead and server side want to remove the "cache" entry.

However, there is another mechanism that can "Re-activate" the client, that is when the server side receives a heartbeat from the client, it will re-activate it.

So that is what you observed, the log say client is removed for a short period of time, and then it re-activated.

How to achieve your main goal, you can do this in the real world setup, use ps -aux | grep 'nvflare' and find the process id for one of the client site.
Then you just do "kill -9 [pid]", this should do your trick.