food vision big feature extraction taking an insanely long amount of time to train

[Donzark]

Even after using all the better performance policies you used in the video and mixed precision the training time is still showing an insanely large ETA. I've tried a number of other architectures from tf.applications as the base model but all of them are showing even larger ETA. I've been stuck for a couple of days now and i really need help.
This is what i'm talking about
3/2368 [..............................] - ETA: 368:00:45 - loss: nan - accuracy: 0.0000e+00

[MimaDalix]

I am having the same problem (ETA: 93:00:20) after 3h of training. did anyone come up with an answer ?

[Donzark]

The problem is that the GPU that u we're allocated doesn't support mixed precision although it is a Tesla t4, I too am wondering why. But if you turn off mixed precision it will work normally although not as fast as it would have been with mixed prescription

…

On Sun, Nov 13, 2022, 8:01 PM Maroua K. ***@***.***> wrote: I am having the same problem (ETA: 93:00:20) after 3h of training. did anyone come up with an answer ? — Reply to this email directly, view it on GitHub <#463 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUDX6CXFYROX2GSKX7SFT2DWIE3KJANCNFSM6AAAAAARSF7QNI> . You are receiving this because you authored the thread.Message ID: <mrdbourke/tensorflow-deep-learning/repo-discussions/463/comments/4129706@ github.com>