[Q&A] Real-World CIFAR-10: No module named 'nvflare.app_common.executors.model_learner_executor'

[manzurelahi]

Python version (python3 -V)

3.10.12

NVFlare version (python3 -m pip list | grep "nvflare")

2.3.8

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

main

Operating system

minimal-22_04-lts-gen2

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Hi
I am trying to learn and run the Real-World Federated Learning with CIFAR-10:
However, I am getting ModuleNotFoundError: No module named 'nvflare.app_common.executors.model_learner_executor'
I am happy to provide more info as needed. Thanks for your assistance.
My Scenario:

1. I run in an Azure VM Ubuntu, minimal-22_04-lts-gen2, Standard NV4as v4. The Azure VM has GPU, but I am facing the challenge installing or utilising it.
2. Python 3.10.12, NVFlare 2.3.8, Phyton virtual environment: nvflare-env
3. No GPU: attached modified meta.json
4. "min_clients": 2 and "num_rounds": 10. Modified config_fed_server.json
5. I followed steps 1, 2, 3.1 with 2 sites, changed secure_project.yml, skipped 3.2 (no GPU)
6. In step 3.3, I started FL system with 2 two clients: ./start_fl_secure.sh 2 and skipped all other remaining steps at 3
7. I am more interested in Secure aggregation using homomorphic encryption, so I have run straight step 4.3: ./submit_job.sh cifar10_fedavg_he 1.0
8. Then the job fails with the error: No module named 'nvflare.app_common.executors.model_learner_executor'. I have attached the log from NVFlare/examples/advanced/cifar10/cifar10-real-world/workspaces/secure_workspace/site-1/dab538dc-66eb-4e06-8145-e98dbd4e236d/log.txt
9. The model_learner_executor.py exists here: NVFlare/nvflare/app_common/executors
10. echo $PYTHONPATH: /home/melahi/NVFlare/examples/advanced/cifar10/cifar10-real-world/.
11. Error:
    File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/utils/json_scanner.py", line 98, in _do_scan node.processor.process_element(node) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 135, in process_element self.process_config_element(self.config_ctx, node) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/client/client_json_config.py", line 100, in process_config_element self.current_exe.executor = self.authorize_and_build_component(element, config_ctx, node) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/private/json_configer.py", line 104, in authorize_and_build_component return self.build_component(config_dict) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/private/fed/client/client_json_config.py", line 104, in build_component t = super().build_component(config_dict) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/utils/component_builder.py", line 86, in build_component return instantiate_class(class_path, class_args) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 51, in instantiate_class c = get_class(class_path) File "/home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/nvflare/fuel/utils/class_utils.py", line 30, in get_class module_ = importlib.import_module(module_name) File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked ModuleNotFoundError: No module named 'nvflare.app_common.executors.model_learner_executor'

[YuanTingHsieh]

@manzurelahi thank you very much for your interests and the questions!

Please use branch "2.3" for NVFlare version 2.3.8. (you can git checkout 2.3 to switch branch)
Let us know if that solves your problem.

[manzurelahi]

Appreciate your quick response. I've successfully switched to branch 2.3, and the job is now underway. Could you guide me on how to determine when the job is complete?

Additionally, I've initiated the job mentioned in step 4.2 using the command ./submit_job.sh cifar10_fedavg_stream_tb 1.0. However, I'm encountering issues with the 'download_job [JOB_ID]' command, as it's not recognized in the same console. Any insights on how to download the job results? Do I need to halt the process in step 4.2 before proceeding to step 4.3? If so, could you provide guidance on stopping the job?

Lastly, for step 5, how can I view the results? Your assistance is highly appreciated!

[YuanTingHsieh]

@manzurelahi glad it is running for you!

If you are following instructions in https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/cifar10/cifar10-real-world
You can start the "admin client console" like the following:

yuantingh@642a59d-lcedt:~/NVFlare/examples/advanced/cifar10/cifar10-real-world$ ./workspaces/workspace/secure_project/prod_00/[email protected]/startup/fl_admin.sh

Inside the admin console, you can issue a lot of commands like "check_status server":

> check_status server
Engine status: stopped
---------------------
| JOB_ID | APP NAME |
---------------------
---------------------
Registered clients: 0 

Or "list_jobs":

> list_jobs 
No jobs found.

Please refer to the doc: https://nvflare.readthedocs.io/en/stable/real_world_fl/operation.html#admin-command-prompt

[manzurelahi]

Thanks for your reply. I have tried,
yuantingh@642a59d-lcedt:~/NVFlare/examples/advanced/cifar10/cifar10-real-world$ ./workspaces/workspace/secure_project/prod_00/[email protected]/startup/fl_admin.sh

and for user name: admin, I am getting Incorrect user name or certificate:

2023-12-09 07:26:09.651837: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2023-12-09 07:26:09.651891: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2023-12-09 07:26:09.653151: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-12-09 07:26:09.659998: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-12-09 07:26:10.445847: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT User Name: admin Trying to obtain server address Obtained server address: localhost:8103 Trying to login, please wait ... Incorrect user name or certificate

Also How can I see see the result in plot as per step 5 and 5.1? Sorry the instruction is not very clear for me.
Thanks for your assistance.

[YuanTingHsieh]

@manzurelahi thanks for getting back to us!

Your user name should not be "admin" but instead should be "[email protected]" in this example.
How do we find that?, we can check the "secure_project.yml" to figure that out: https://github.com/NVIDIA/NVFlare/blob/2.3/examples/advanced/cifar10/cifar10-real-world/workspaces/secure_project.yml#L41

How to see step 5 and step 5.1 results?

1. You need to first figure out your "JOB ID" using "list_jobs" admin commands.
2. And then you can download the job results using "download_job [job id]"
3. Then you can run the script "python ./figs/plot_tensorboard_events.py"

@holgerroth @SYangster we might want to enhance this README for better user experiences.

[chesterxgchen]

For 2.3 branch, the POC was using POC mode, the user name is "admin". in 2.4.0, POC is using production mode, the POC admin console user is "[email protected]"

I think the user might have used the 2.4.0 POC command created the POC workspace and startup kit, then switch back to 2.3 branch.

To be able to follow the example, make sure clean the workspace and re-run the POC command create a workspace that match the branch

[manzurelahi]

Thanks for your guidance. I have made some progress.

1. While the job finished after 3 hours (./submit_job.sh cifar10_fedavg_he 1.0), I logged in to the admin console and downloaded the job: download_job [job id]
2. Started another terminal export PYTHONPATH=${PWD}/..and tried: python ./figs/plot_tensorboard_events.py
3. Error:
   Traceback (most recent call last): File "/home/melahi/NVFlare/examples/advanced/cifar10/cifar10-real-world/./figs/plot_tensorboard_events.py", line 145, in <module> main() File "/home/melahi/NVFlare/examples/advanced/cifar10/cifar10-real-world/./figs/plot_tensorboard_events.py", line 112, in main job_id = find_job_id(workdir=download_dir, fl_app_name=config_name, alpha=alpha) File "/home/melahi/NVFlare/examples/advanced/cifar10/cifar10-real-world/./figs/plot_tensorboard_events.py", line 67, in find_job_id raise ValueError(f"No job id found for fl_app_name={fl_app_name} in workdir={workdir}")

[manzurelahi]

Progress update:
Looks like failing at plot_tensorboard_events.py line 112 job_id = find_job_id(workdir=download_dir, fl_app_name=config_name, alpha=alpha)
I have manually set job_id = "ac4f6338-59a2-4c22-9a10-8f69ac6fa9ad"
I am getting results but not getting any interactive graph:
2023-12-14 10:24:26.139309: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2023-12-14 10:24:26.139359: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2023-12-14 10:24:26.140549: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2023-12-14 10:24:26.147155: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-12-14 10:24:26.870243: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Found run ac4f6338-59a2-4c22-9a10-8f69ac6fa9ad for cifar10_fedavg_stream_tb with alpha=1.0 adding ./workspaces/secure_workspace/site-1/ac4f6338-59a2-4c22-9a10-8f69ac6fa9ad/app_site-1/events.out.tfevents.1702525627.VM-Ubuntu-LTS.1372.0 WARNING:tensorflow:From /home/melahi/NVFlare/nvflare-env/lib/python3.10/site-packages/tensorflow/python/summary/summary_iterator.py:27: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version. Instructions for updating: Use eager execution and: tf.data.TFRecordDataset(path) added 10 entries for val_acc_global_model Found run ac4f6338-59a2-4c22-9a10-8f69ac6fa9ad for cifar10_fedavg_he with alpha=1.0 adding ./workspaces/secure_workspace/site-1/ac4f6338-59a2-4c22-9a10-8f69ac6fa9ad/app_site-1/events.out.tfevents.1702525627.VM-Ubuntu-LTS.1372.0 added 10 entries for val_acc_global_model Training TB data: Config Step Accuracy 0 cifar10_fedavg_stream_tb 0 0.1000 1 cifar10_fedavg_stream_tb 3 0.1025 2 cifar10_fedavg_stream_tb 7 0.4426 3 cifar10_fedavg_stream_tb 11 0.6118 4 cifar10_fedavg_stream_tb 15 0.6769 5 cifar10_fedavg_stream_tb 19 0.7012 6 cifar10_fedavg_stream_tb 23 0.7519 7 cifar10_fedavg_stream_tb 27 0.7955 8 cifar10_fedavg_stream_tb 31 0.8184 9 cifar10_fedavg_stream_tb 35 0.8237 10 cifar10_fedavg_he 0 0.1000 11 cifar10_fedavg_he 3 0.1025 12 cifar10_fedavg_he 7 0.4426 13 cifar10_fedavg_he 11 0.6118 14 cifar10_fedavg_he 15 0.6769 15 cifar10_fedavg_he 19 0.7012 16 cifar10_fedavg_he 23 0.7519 17 cifar10_fedavg_he 27 0.7955 18 cifar10_fedavg_he 31 0.8184 19 cifar10_fedavg_he 35 0.8237 Cross-site val data: Config SRV_FL_global_model.pt SRV_best_FL_global_model.pt 0 cifar10_fedavg_stream_tb 0.8324 0.8237 1 cifar10_fedavg_he 0.8324 0.8237 /home/melahi/NVFlare/examples/advanced/cifar10/cifar10-real-world/./figs/plot_tensorboard_events.py:148: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown plt.show()

[YuanTingHsieh]

@manzurelahi thanks for updating, maybe @holgerroth can help comment on the drawing part thanks!

[holgerroth]

Hi @manzurelahi, the figure doesn't show because of UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown plt.show(). Where are you running this? In the terminal or IDE like VSCode or PyCharm? I see this issue sometimes when running in an IDE. Please try the terminal in this case. There might be some hints here on how to solve the issue.

[manzurelahi]

Hi @holgerroth, Thanks for your reply.
I am running in the Ubuntu terminal and still having issues seeing the result.
The Job takes more than 3 hours to finish. The server is now restarted, and no job is running.
Do I need to rerun the experiments/jobs to see the resulting plot? Can I use the existing results (finished yesterday) to create the plot?

[holgerroth]

Yes, the plots are created from parsing the tensorboard files. As long as you have kept your output folder from yesterday, you can read those files. Try to use tensorboard first to see if all the results are there.

Another thing to try is to use a clean python virtual environment and install all the requirements.txt from this example there. Otherwise, you can try to change the matplotlib backend https://stackoverflow.com/questions/3285193/how-to-change-matplotlib-backends