pytorch_lightning.utilities.exceptions.MisconfigurationException: You requested gpu: [0, 1, 2, 3, 4, 5, 6, 7] But your machine only has: [0]

[marcelbra]

Hello, I want to run my training procedure in parallel, however, get

pytorch_lightning.utilities.exceptions.MisconfigurationException: You requested gpu: [0, 1, 2, 3, 4, 5, 6, 7]
 But your machine only has: [0]

When I check for GPUs I get

>>> print(torch.__version__)
1.12.1+cu102
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
8

None of them are being used at the time. What could be the problem? I found similar problems related to this, but none could resolve my problem and usually when someones asked no GPUs were detected at all.

I'm calling the trainer like so:

trainer = pl.Trainer(accelerator="gpu",
                       devices=8,
                       strategy="ddp",
                       default_root_dir=model_save_dir,
                       callbacks=[checkpoint_callback],
                       #gpus=list(range(1,8)),
                       max_epochs=args.max_epochs,
                       gradient_clip_val=5,
                       logger=wandb_logger,
                       log_every_n_steps=100,
                       limit_val_batches=0)

I'm on Ubuntu 20.04.5 LTS using Python 3.8.10.

What other information do you need?

/edit: Just noticed, maybe this is more appropriate in the ddp forum.

[marcelbra]

I found my bug, it's something very random, probably doesn't help anyone if I describe what happened. Please close.

[chenaoxuan]

Hello, can you explain how to solve this problem? thank!

[aleksmirosh]

I have the same issue
could you please describe your solution? @marcelbra

[chenaoxuan]

I tried to locate the problem, there is a method named "parser_gpu_ids()" in "pytorch_ lightning.utilities.device_parse"

This method automatically obtains all currently available GPUs by calling the "_sanitize_gpu_ids()" method. If the GPU you specify is not included, an exception will be thrown;

What shocked me was that this method called pytorch's API: "torch. cuda. device_count()"; But everything in Pytorch is normal....

[chenaoxuan]

Can you share your solution?

Thanks!!!!!!!!!

[aleksmirosh]

@chenaoxuan
for me the problem was next:
Inside my main script I called import from the script where I made visible only 1 gpu a while ago.
I called just one function, so I did not expect it will influence my main script, but it did.

[chenaoxuan]

I am trying to solve this problem. Perhaps there is an issue with the environment variable, but who knows...

[jxtngx]

Can each of you run the following in terminal please

echo $CUDA_VISIBLE_DEVICES

[jxtngx]

@chenaoxuan which version of PL or lightning are you using?

[orena1]

I mistakenly used tasks-per-node instead of ntasks-per-node, if that helps.