RuntimeError: cublas runtime error : the GPU program failed to execute

[JanKybe]

Have anyone gotten that kind of error while sampling

Loading prior in eval mode
Traceback (most recent call last):
  File "jukebox/sample.py", line 220, in <module>
    fire.Fire(run)
  File "C:\Users\user\miniconda3\envs\jbox\lib\site-packages\fire\core.py", line 127, in Fire
    component_trace = _Fire(component, args, context, name)
  File "C:\Users\user\miniconda3\envs\jbox\lib\site-packages\fire\core.py", line 366, in _Fire
    component, remaining_args)
  File "C:\Users\user\miniconda3\envs\jbox\lib\site-packages\fire\core.py", line 542, in _CallCallable
    result = fn(*varargs, **kwargs)
  File "jukebox/sample.py", line 217, in run
    save_samples(model, device, hps, sample_hps)
  File "jukebox/sample.py", line 205, in save_samples
    primed_sample(x, labels, sampling_kwargs, priors, hps)
  File "jukebox/sample.py", line 142, in primed_sample
    zs = priors[-1].encode(x, start_level=0, end_level=len(priors), bs_chunks=x.shape[0])
  File "c:\users\user\documents\jukebox\data\jukebox\prior\prior.py", line 210, in encode
    zs = self.encoder(x, start_level=start_level, end_level=end_level, bs_chunks=bs_chunks)
  File "c:\users\user\documents\jukebox\data\jukebox\vqvae\vqvae.py", line 139, in encode
    zs_i = self._encode(x_i, start_level=start_level, end_level=end_level)
  File "c:\users\user\documents\jukebox\data\jukebox\vqvae\vqvae.py", line 132, in _encode
    zs = self.bottleneck.encode(xs)
  File "c:\users\user\documents\jukebox\data\jukebox\vqvae\bottleneck.py", line 192, in encode
    zs = [level_block.encode(x) for (level_block, x) in zip(self.level_blocks, xs)]
  File "c:\users\user\documents\jukebox\data\jukebox\vqvae\bottleneck.py", line 192, in <listcomp>
    zs = [level_block.encode(x) for (level_block, x) in zip(self.level_blocks, xs)]
  File "c:\users\user\documents\jukebox\data\jukebox\vqvae\bottleneck.py", line 132, in encode
    x_l, fit = self.quantise(x)
  File "c:\users\user\documents\jukebox\data\jukebox\vqvae\bottleneck.py", line 115, in quantise
    distance = t.sum(x ** 2, dim=-1, keepdim=True) - 2 * t.matmul(x, k_w) + t.sum(k_w ** 2, dim=0,
RuntimeError: cublas runtime error : the GPU program failed to execute at C:/w/1/s/tmp_conda_3.7_044431/conda/conda-bld/pytorch_1556686009173/work/aten/src/THC/THCBlas.cu:259

[rmarchant]

Were you ever able to workaround it? I've just hit this myself.

[jimouris]

@rmarchant I updated cublas/pytorch and it worked for me: pip install torch===1.7.1+cu110 torchvision===0.8.2+cu110 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html -i https://pypi.douban.com/simple