FIX: Input 'y' of 'Sub' Op has type float16 that does not match type float32 of argument 'x'.

[ashikshafi08]

Temporary Fix

This seems like a bug with the latest version of TensorFlow (2.5.0). In case if you guys want to monitor the thread check this link.

For now, there is a temporary fix for this that is to downgrade your TensorFlow version to 2.4.1 to avoid this error. We should stick with this for the time being till the developers in TensorFlow find a way to fix this.

# Downgrade tensorflow version (on your colab notebook) 
!pip install tensorflow==2.4.1 
 
# If you are using Anaconda (on your PC) 
conda install tensorflow==2.4.1

[mrdbourke]

Thank you for sharing brother! I'll add this to the course content.

PS the main place this will happen is for Notebook 07 - Milestone Project 1: Food Vision when we use tf.keras.mixed_precision to train any EfficientNetBX models.

[mrdbourke]

I wrote a note about this on the ZTM/Udemy versions of the course, putting it here for reference.

It comes just before the mixed precision video for Notebook 07 - Milestone Project 1: Food Vision.

---

Hey there,

Just putting this note here to let you know that you may face an error in the next video if you're using TensorFlow 2.5+.

Error

The error happens when you turn on mixed precision training for EfficientNetBX models (if you're not sure what this is, you'll see it in the next video).

It'll look something like this:

TypeError: Input 'y' of 'Sub' Op has type float16 that does not match type float32 of argument 'x'

As of 29 May 2021 it looks like this is a bug in TensorFlow 2.5+.

There is an issue thread tracking the progress of an update on GitHub: tensorflow/tensorflow#49725

How to fix it

The current workaround is to downgrade to TensorFlow 2.4.1 (the last version before 2.5).

You can do this in Google Colab by running:

# Downgrade Tensorflow Version (run this in Google Colab) 
!pip install tensorflow==2.4.1

After doing so, you'll have to restart your runtime (Runtime -> Restart runtime) to ensure TensorFlow 2.4.1 is active.

# Check if TensorFlow 2.4.1 is active 
import tensorflow as tf
tf.__version__

>>> '2.4.1'

Once TensorFlow 2.4.1 is installed, the code should work as normal.

If you have any issues, please post your question/reply below so others can see.

If a fix for this issue comes for TensorFlow 2.5+ (it should), we'll update this post.

Happy mixed precision training,

• Daniel

[jrassoul]

Hello Daniel, Hello everyone.
System information

• Windows 11
• I use Jupyter Notebook.
• TensorFlow Version: 2.9.0
• Keras Version: 2.9.0
• GPU 0: NVIDIA GeForce GTX 1650

No problem with Mixed precision compatibility check but I had a problem with EfficientNetB0
base_model = tf.keras.applications.EfficientNetB0(include_top=False)

Solution :

• I installed this package !pip install --user tf-nightly
• After this I had a warning to copy the path in path system, the path is "C:/Users/user_name/AppData/Roaming/Python/Python39/Scripts".
• Restart the environment.

It worked for me, yeeeeees.

This article may also interest you : What is the difference between tf-nightly and tensorflow in PyPI?

Good continuation.
Juba

[Jayms8462]

Also running into a problem. It appears that the ETA for fitting the model after downgrading to 2.4.1 is running at like 100 hours even with using the Nvida T4. Even running the project directly from github https://github.com/mrdbourke/tensorflow-deep-learning/blob/main/07_food_vision_milestone_project_1.ipynb it still is getting 100+ hours

Saving TensorBoard log files to: training_logs/efficientnetb0_101_classes_all_data_feature_extract/20221227-085937
Epoch 1/3
3/2368 [..............................] - ETA: 97:10:20 - loss: nan - accuracy: 0.0382

Tue Dec 27 08:59:33 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 66C P0 31W / 70W | 3MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

Turn on mixed precision training

from tensorflow.keras import mixed_precision
mixed_precision.set_global_policy(policy="mixed_float16") # set global policy to mixed precision

WARNING:tensorflow:Mixed precision compatibility check (mixed_float16): WARNING
The dtype policy mixed_float16 may run slowly because this machine does not have a GPU. Only Nvidia GPUs with compute capability of at least 7.0 run quickly with mixed_float16.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once

[Jayms8462]

I left this running for a few hours in case the ETA was not correct.
Running this on google colab, the GPU is enabled in the runtime as seen above and I am using the paid middle tier. I also confirmed the GPU as a NVIDIA T4 right before fitting the model.

[Jayms8462]

Just an Update, I was able to run mixed_precision in Tensorflow 2.11 with no issue

[Jayms8462]

I was also able to get passed my problem by running Juniper Notebook locally on my system with a 3060ti and 64 GB of ram. This sped up the fitting of my model with all the data. instead of getting a few hours executing the fit, it dropped down to 2 min on the first epoch.

The solution to the issue is just updating to TensorFlow 2.11

[eMIKARp]

Had the same problem.
Thanks for your suggestion!

[eMIKARp]

This should probably be included in the note Daniel has created about the original issue with mixed precision.