EfficientNetV2

[LinusJ79]

Is it possible to use EfficientNetV2 in a functional model and do feature extraction and then fine tuning?
So far i am only able to do this with EfficientNetBX from keras.applications.

[mrdbourke]

Yes, you'll want to turn the model into a hub.KerasLayer(), see this guide for more: https://www.tensorflow.org/tutorials/images/transfer_learning_with_hub

[mrdbourke]

You can also see a reference notebook I created using EfficientNetV2 here: #166 (comment)

[LinusJ79]

I dont think you can access the layers using hub.KerasLayers(). If i remember correctly that was the error message i got.
When i am about to fine tune and set some layers trainable that is not possible with hub.KerasLayers(), only with keras.applications EffNet V1. Or do you know how to work around this and access the layers?

[mrdbourke]

Hmm... I'm not sure about this one.

Can you post your error here?

To fine-tune, you can set trainable=True:

 # Download the pretrained model and save it as a Keras layer
 feature_extractor_layer = hub.KerasLayer(model_url,
                                           trainable=True, # make layers trainable for fine-tuning
                                           name='feature_extraction_layer',
                                           input_shape=(224, 224, 3)) # define the input image shape

The code above is from the linked notebook above.

[LinusJ79]

I want to feature extract AND then fine-tune. Is it something like this you are suggesting?:

def create_model(model_url, NUM_CLASSES=243, IMG_SHAPE=IMG_SHAPE):
  
  # Download the pretrained model and save it as a Keras layer
  feature_extractor_layer = hub.KerasLayer(model_url,
                                           trainable=False, # freeze the underlying patterns
                                           name='feature_extraction_layer',
                                           input_shape=IMG_SHAPE) # define the input image shape
  # Create our own model
  inputs = tf.keras.layers.Input(shape=IMG_SHAPE)
  x = tf.keras.layers.Rescaling(1./255)(inputs)
  x = feature_extractor_layer(x)
  x = tf.keras.layers.Dense(NUM_CLASSES)(x)  
  
  outputs = tf.keras.layers.Activation("softmax", dtype=tf.float32, name="softmax_float32")(x) 

  model = tf.keras.Model(inputs, outputs)

  return mode

import tensorflow_hub as hub
from tensorflow.keras import layers
# EfficientNetB0 v2 feature vector
efficientnetb0_v2_url = "https://tfhub.dev/google/imagenet/efficientnet_v2_imagenet21k_ft1k_s/feature_vector/2"

# Create EfficientNetB0 V2 model
efficientnetb0_v2_ = create_model(model_url=efficientnetb0_v2_url, # use EfficientNetB0 TensorFlow Hub URL
                                 NUM_CLASSES=243)

# Compile EfficientNetB0 V2 model
efficientnetb0_v2_.compile(loss='sparse_categorical_crossentropy',
                           optimizer=tf.keras.optimizers.Adam(),
                           metrics=['accuracy'])

for layer_number, layer in enumerate(efficientnetb0_v2_.layers):
  print(layer_number, layer.name, layer.trainable)`

0 input_3 True
1 rescaling_2 True
2 feature_extraction_layer False
3 dense_1 True
4 softmax_float32 True

# Fit EfficientNetB0 V2 model 
efficientnetb0_v2_history = efficientnetb0_v2_.fit(train_dataset,
                                                  epochs=1, # train for 10 epochs
                                                  validation_data=val_dataset)`

457/457 [==============================] - 365s 738ms/step - loss: 2.4104 - accuracy: 0.4908 - val_loss: 1.6370 - val_accuracy: 0.6322

def unfreeze_model(model):

    fine_tuning_layer = hub.KerasLayer(model_url = efficientnetb0_v2_url,
                                           trainable=True, # freeze the underlying patterns
                                           name='fine_tuning_layer',
                                           input_shape=IMG_SHAPE) # define the input image shape
   
    model.layers[2] = fine_tuning_layer
    model.compile(loss='sparse_categorical_crossentropy',
                           optimizer=tf.keras.optimizers.Adam(),
                           metrics=['accuracy'])

    return model

eff_fine_v2_model = unfreeze_model(efficientnetb0_v2_)

# Check which layers are tuneable in the whole model
for layer_number, layer in enumerate(eff_fine_v2_model.layers):
  print(layer_number, layer.name, layer.trainable)`

0 input_3 True
1 rescaling_2 True
2 feature_extraction_layer False
3 dense_1 True
4 softmax_float32 True

Notice how

fine_tuning_layer = hub.KerasLayer(model_url,
                                           trainable=True, # freeze the underlying patterns
                                           name='fine_tuning_layer',
                                           input_shape=IMG_SHAPE) # define the input image shape
   
    model.layers[2] = fine_tuning_layer

does not change the model and it is not trainable, i.e. no fine tuning.