CUDA solver failure

[andreaferretti]

Hi, I tried the CelebA example from the README, using your environment and the provided images, but I get the error

ERROR - main.py - 2022-06-17 10:28:39,626 - Traceback (most recent call last):
  File "/home/wizard/dev/ddrm/main.py", line 164, in main
    runner.sample()
  File "/home/wizard/dev/ddrm/runners/diffusion.py", line 163, in sample
    self.sample_sequence(model, cls_fn)
  File "/home/wizard/dev/ddrm/runners/diffusion.py", line 237, in sample_sequence
    H_funcs = SuperResolution(config.data.channels, config.data.image_size, blur_by, self.device)
  File "/home/wizard/dev/ddrm/functions/svd_replacement.py", line 179, in __init__
    self.U_small, self.singulars_small, self.V_small = torch.svd(H, some=False)
RuntimeError: cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSgesvdj( handle, jobz, econ, m, n, A, lda, S, U, ldu, V, ldv, static_cast<float*>(dataPtr.get()), lwork, info, params)`

Probably there is a mismatch between the CUDA versions. Which CUDA did you use for your experiments?

[bahjat-kawar]

Hello Andrea,
We used CUDA version 11 in our experiments. Please refer to the provided environment file for more details.

[andreaferretti]

In case anyone else incurs in this issue: it is related to a combination of CUDA version installed locally and the CUDA version used in the provided environment file. See here for details