04 Custom Dataset: Error if num_workers > 0 in DataLoader

[fivefishstudios]

Error Message is:
DataLoader worker (pid 52116) exited unexpectedly with exit code 1. Details are lost due to multiprocessing.
Rerunning with num_workers=0 may give better error trace.

If I set CPU_COUNT = 0, so that num_workers=0 in DataLoader, everything works without error.

Any ideas?

OSX 10.15.7. Python 3.8.8, PyTorch 1.13.0

[fivefishstudios]

Welp, I guess I solved it, thanks to Google and Github.
I don't understand why/how that fixed the problem.

SOLUTION:

# add this import
from joblib.externals.loky.backend.context import get_context

BATCH_SIZE = 32
CPU_COUNT = os.cpu_count()    # this now can be > 0

# add multiprocessing_context in DataLoader
train_dataloader = DataLoader(
    dataset=train_data,
    batch_size=BATCH_SIZE,
    shuffle=True,
    num_workers=CPU_COUNT,
    multiprocessing_context=get_context('loky') # ADD THIS LINE
)

[mrdbourke]

Woah, glad to hear you got it fixed.

I've never seen that solution but if it works, that's epic!

PyTorch + multiprocessing (and multiprocessing in general) is a deep topic.

[prinkeeras]

Thank you! I was having the same issue using jupyter notebook in vscode on windows. This solution fixed it for me!

[PawKanarek]

Im on intel macbook2018, and i don't have an error, but the iterating next(iter(train_dataloader_simple)) is very slow. Setting num_workers adds a delay of about 8 seconds on my machine while iterating through the DataLoader. And training time goes from seconds to minutes.

Some more info. Setting parameter num_workers=0 on DataLoader makes the problem go away.
More info: https://discuss.pytorch.org/t/dataloader-super-slow/38686

Edit: Im using local Jupyter notebooks :)