07 - Food Vision Big - evaluating the model

[pawelkiszczak]

Dear colleagues,
I stumbled upon as issue while trying to evaluate the fine-tuned model.

As I did the whole training sequence and wanted to evaluate the metrics that were mentioned in the solutions notebook, some of them (like a confusion matrix or typical Sklearn metrics: recall, f1-score, precision) require an array/tensor of truth labels. However, as the data is stacked and held in prefetched batches it's not so obvious to extract desired output (or I do have not-so-great ideas for doing this). Or my model is so good that it can break the threshold of 100% accuracy and end up with about 500%

Cutting to the chase, I've tried a couple of methods:

• using unbatch() with tuple unpacking gave each time different order of labels
• TensorArray also results in different output each time I use it

What did match is the length of output from the aforementioned solutions was equal to the number of predictions/test images. I'd like to also add that I didn't change anything in batching or prefetching section so data should not be shuffled (or our friendly TF does not tell this to us).

Hoping I've made my description clear enough. If not, let me know what more can I provide to solve this very irritating issue.

EDIT:
@mrdbourke - I'd be very grateful if you could help on this one

Thanks!

[ppmarkus]

Hi Pawel, your method to extract the true labels using unbatch() is correct. It's just that your dataset is shuffled, which is why the values come out in a different order every time. it's a bit confusing I guess because Dan doesn't do this in the vids.

As was covered in section section before, you'll need to turn off shuffle when reading the dataset

(train_data, test_data), ds_info = tfds.load('food101',
split=['train', 'validation'],
shuffle_files=False, # <---- here
as_supervised=True,
with_info=True)

Then when setting up the pipeline, onlly shuffle the training set and not the test set.

set up the pipeline for training

train_data = train_data.map(map_func=preprocess_img,num_parallel_calls=tf.data.AUTOTUNE).shuffle(buffer_size=1000).batch(batch_size=32).prefetch(buffer_size=tf.data.AUTOTUNE)

do the same for test data but don't need to shuffle

test_data = test_data.map(map_func=preprocess_img, num_parallel_calls=tf.data.AUTOTUNE).batch(batch_size=32).prefetch(buffer_size=tf.data.AUTOTUNE)

[skdarkey]

I am having similar issues like Pawel. Everything works fine until I get to the using the sci-kit learn accuracy method. It brings zero accuracy all the time. After turning shuffle to false as noted in your response, then seems to be no difference. I am now observing that after unbatching the test data, it is bringing only one class, which is class zero. Which is why trying to compare it with predicted classes is bringing zero accuracy. Do you have any more insights into why this is happening?.