[Q&A] How to improve accuracy on global model during Aggregation process?

[plaipmc]

Python version (python3 -V)

3.10.12

NVFlare version (python3 -m pip list | grep "nvflare")

2.4.0

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.4.0rc2

Operating system

Ubuntu 22.04.3

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

I trying to compare the performance between Centralized Learning and Federated Learning using neural network with PyTorch model. (The dataset is the Numerical data)

I create the job using sag_pt job template and split dataset to each client. After running some simulation, I bring the global model in simulate_job\app_server\FL_global_model.pt to test using each client test data and the dataset which combine all the test data from all clients. Eventhough the validation metric (which I assume and think that validation metric is the accuracy of each client model ) is good (0.8 on average), but the accuracy in testing is around 0.5 which didn't be like validation metric. More than that, I bring some client model to test and got around 0.8 on their test data and around 0.55 on other dataset which is really make sense.

So I assume that the Aggregation method would be the key in this situation. Right now, I use only FedAvg which is the default of the job template and each client have weight 1.0 (according to the default settings).

Do you have some recommendation for me in this situation? Should I try another aggregation method? and how to try it?

[chesterxgchen]

@holgerroth can you make suggestions

[YuanTingHsieh]

@plaipmc

Since you are doing experiment, I will suggest you start from regular centralized learning for tuning your parameters.

When putting all data into one, split it into train set, validation set and testing set.
Try to use the validation set to determine the best parameters for your algorithm.

Now, comes to the federated learning part, once you get those good parameters for a centralized learning, you can do:
Split the data into each site.
For each site split their assigned data into train, validation and testing.
Use the training dataset and validation dataset to determine when we reach a good performance.
Use the FL_Global_model to test for each testing data.
Each client has a aggregation weight of 1.0 is good.

I would say the result should be very close to the centralized training ^