Failure to load a model created with v1.5.0 in v1.5.1

[dzenanz]

Describe the bug
I have a code based on https://github.com/Project-MONAI/tutorials/blob/main/2d_segmentation/torch/unet_training_dict.py
at revision 8b90a16d82a463551d81dd4da200f3d406c6843e. Key code parts:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = UNet(
    spatial_dims=2,
    in_channels=1,
    out_channels=1,
    channels=(16, 32, 64, 128, 256),
    strides=(2, 2, 2, 2),
    num_res_units=2,
).to(device)

torch.save(model.state_dict(), "cancer_model.pth")  # during training
model.load_state_dict(torch.load("cancer_model.pth", weights_only=True))  # in evaluate

I recently upgraded my MONAI and PyTorch versions:

and now the old model cannot be loaded by the evaluator:

M:\Dev\PSI\PathologyAnalysis\.venv\Scripts\python.exe M:\Dev\PSI\PathologyAnalysis\evaluate.py M:\Dev\PSI\PathologyAnalysis\Images 
Using image list from M:\Dev\PSI\PathologyAnalysis\Images. Working directory: M:\Dev\PSI\PathologyAnalysis\Images\Temp
Traceback (most recent call last):
  File "M:\Dev\PSI\PathologyAnalysis\evaluate.py", line 66, in <module>
    main(sys.argv[1])
  File "M:\Dev\PSI\PathologyAnalysis\evaluate.py", line 32, in main
    model.load_state_dict(torch.load("cancer_model_1.pth", weights_only=True))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 1521, in load
    return _load(
           ^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 2119, in _load
    result = unpickler.load()
             ^^^^^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\_weights_only_unpickler.py", line 532, in load
    self.append(self.persistent_load(pid))
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 2083, in persistent_load
    typed_storage = load_tensor(
                    ^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 2049, in load_tensor
    wrap_storage = restore_location(storage, location)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 698, in default_restore_location
    result = fn(storage, location)
             ^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 636, in _deserialize
    device = _validate_device(location, backend_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\torch\serialization.py", line 605, in _validate_device
    raise RuntimeError(
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

Process finished with exit code 1

Is this expected?

Environment

================================
Printing MONAI config...
================================
MONAI version: 1.5.1
Numpy version: 2.1.2
Pytorch version: 2.8.0+cpu
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 9c6d819f97e37f36c72f3bdfad676b455bd2fa0d
MONAI __file__: M:\Dev\PSI\PathologyAnalysis\.venv\Lib\site-packages\monai\__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.11
ITK version: 5.4.4
Nibabel version: 5.3.2
scikit-image version: 0.25.2
scipy version: 1.16.2
Pillow version: 11.0.0
Tensorboard version: 2.20.0
gdown version: 5.2.0
TorchVision version: 0.23.0+cpu
tqdm version: 4.67.1
lmdb version: 1.7.3
psutil version: 7.0.0
pandas version: 2.3.2
einops version: 0.8.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: 3.3.2
pynrrd version: 1.1.3
clearml version: 2.0.3rc0

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Windows
Win32 version: ('11', '10.0.26100', 'SP0', 'Multiprocessor Free')
Win32 edition: Professional
Platform: Windows-11-10.0.26100-SP0
Processor: AMD64 Family 25 Model 33 Stepping 0, AuthenticAMD
Machine: AMD64
Python version: 3.12.5
Process name: python.exe
Command: ['C:\\Program Files\\Python312\\python.exe', 'M:\\Dev\\PSI\\PathologyAnalysis\\evaluate.py', 'M:\\Dev\\PSI\\PathologyAnalysis\\Images']
Open files: [popenfile(path='C:\\Windows\\System32\\en-US\\KernelBase.dll.mui', fd=-1), popenfile(path='C:\\Windows\\System32\\en-US\\kernel32.dll.mui', fd=-1), popenfile(path='C:\\Windows\\System32\\en-US\\tzres.dll.mui', fd=-1)]
Num physical CPUs: 12
Num logical CPUs: 24
Num usable CPUs: 24
CPU usage (%): [11.7, 55.1, 32.1, 33.6, 31.7, 19.0, 13.0, 5.7, 19.3, 20.8, 7.8, 3.6, 4.2, 2.7, 3.3, 1.2, 3.9, 1.2, 2.4, 1.2, 0.6, 2.4, 0.3, 0.6]
CPU freq. (MHz): 3001
Load avg. in last 1, 5, 15 mins (%): [0.0, 0.0, 0.0]
Disk usage (%): 87.8
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 127.9
Available memory (GB): 106.7
Used memory (GB): 21.2

================================
Printing GPU config...
================================
Num GPUs: 0
Has CUDA: False
cuDNN enabled: False
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None

[kavin2003]

Hi @dzenanz,
I might have misunderstood what you were trying to do, but from the environment summary it looks like you’re running on a machine without any GPUs and using a CPU-only build of PyTorch (2.8.0+cpu, Has CUDA: False).
That doesn’t seem to be an issue with MONAI itself — it’s just a device mismatch. Following the suggestion in the error message (to map the model to CPU when loading) should fix it. 🙂

[dzenanz]

This indeed pointed me in the right direction. Are PyTorch 2.6 and 2.8 behaving differently regarding CUDA support? pip install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124 had CUDA support, while pip install torch==2.8.0 --index-url https://download.pytorch.org/whl/cu124 doesn't. I am investigating further.

[dzenanz]

Ah I see now. There is no PyTorch 2.8 which was compiled against CUDA 12.4. So it installed a CPU-only version. If there was a warning, it was buried in a wall of text of other output, and I didn't see it.

[dzenanz]

Installing a proper version of PyTorch fixes the problem. Some care was needed. Thank you for help!