The two Gpus load inconsistent checkpoints after training for a while and saving the previous checkpointq.

[ShuangMa156]

Begin second stage training, cycle  2 ==================
第二阶段:周期2, best_model_path:/root/data1/GEM1/GEM/codes/checkpoint/Ev-REDS-mambaSR/best-stage=2-cycle=01-epoch=029-loss=2.41.ckpt
Load model from  /root/data1/GEM1/GEM/codes/checkpoint/Ev-REDS-mambaSR/best-stage=2-cycle=01-epoch=029-loss=2.41.ckpt
[rank: 0] Seed set to 1226
Begin second stage training, cycle  2 ==================
第二阶段:周期2, best_model_path:/root/data1/GEM1/GEM/codes/checkpoint/Ev-REDS-mambaSR/best-stage=2-cycle=01-epoch=029-loss=2.49.ckpt
Load model from  /root/data1/GEM1/GEM/codes/checkpoint/Ev-REDS-mambaSR/best-stage=2-cycle=01-epoch=029-loss=2.49.ckpt
[rank: 1] Seed set to 1226
Traceback (most recent call last):
  File "/root/data1/GEM1/GEM/codes/main.py", line 289, in <module>
    best_model_path = second_stage_training(args)
  File "/root/data1/GEM1/GEM/codes/main.py", line 76, in second_stage_training
    model = MInterface(**vars(args))
  File "/root/data1/GEM1/GEM/codes/model/model_interface.py", line 37, in __init__
    state_dict = self.load_state_dict_from_checkpoint(kargs['ckpt_path'])
  File "/root/data1/GEM1/GEM/codes/model/model_interface.py", line 183, in load_state_dict_from_checkpoint
    state_dict = torch.load(checkpoint_path)['state_dict']
  File "/root/anaconda3/envs/gem3/lib/python3.10/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/root/anaconda3/envs/gem3/lib/python3.10/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/root/anaconda3/envs/gem3/lib/python3.10/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/root/data1/GEM1/GEM/codes/checkpoint/Ev-REDS-mambaSR/best-stage=2-cycle=01-epoch=029-loss=2.49.ckpt'
GPU device:[0, 1]
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
[rank: 0] Seed set to 1226
[rank: 1] Child process with PID 26229 terminated with code 1. Forcefully terminating all other processes to avoid zombies 🧟
Killed``

save results: