Very Low GPU Utilization During Training with High CPU Load

[yibozhong]

Thank you for your great work. @Andy1621

Describe the issue

During training with K400 dataset, I noticed that the GPU cannot maintain high utilization and often stays idle (low power usage). Meanwhile, the CPU shows consistently high usage. I use num_frames=32, resolution=224.

Suspected cause

The issue appears to be caused by on-the-fly video decoding using decord, which may become a bottleneck and prevent efficient GPU feeding. A relative issue in VideoMAE repo: MCG-NJU/VideoMAE#81

Expected behavior

Idealy, GPU should maintain high utilization during training, with video loading and decoding not becoming the bottleneck.

Some questions

Does preprocessing the video data into tensors help? Have you also observed the same issue during training? If so, how did you solve it? I'd appreciate any advice on this topic. Thank you in advance for your time.