![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "0FJFQ_1h0G3s"
- },
- "source": [
- "# Introduction to TensorFlow, Deep Learning and Transfer Learning (work in progress)\n",
- "\n",
- "* **Project:** Dog Vision πΆπ - Using computer vision to classify dog photos into different breeds.\n",
- "* **Goals:** Learn TensorFlow, deep learning and transfer learning, beat the original research paper results (22% accuracy).\n",
- "* **Domain:** Computer vision.\n",
- "* **Data:** Images of dogs from [Stanford Dogs Dataset](http://vision.stanford.edu/aditya86/ImageNetDogs/) (120 dog breeds, 20,000+ images).\n",
- "* **Problem type:** Multi-class classification (120 different classes).\n",
- "* **Runtime:** This project is designed to run end-to-end in [Google Colab](https://colab.research.google.com/) (for free GPU access and easy setup). If you'd like to run it locally, it will require environment setup.\n",
- "\n",
- "Welcome, welcome!\n",
- "\n",
- "The focus of this notebook is to give a quick overview of deep learning with TensorFlow/Keras.\n",
- "\n",
- "How?\n",
- "\n",
- "We're going to go through the machine learning workflow steps and build a computer vision project to classify photos of dogs into their respective dog breed (a Predictive AI task, see below for more).\n",
- "\n",
- "![\"outline](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-dog-vision-project-outline.png?raw=true\")
\n",
- "\n",
- "*What we're going to build: Dog Vision πΆποΈ, a neural network capable of identifying different dog breeds in images. All the way from dataset preparation to model building, training and evaluation.*"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "n8IXkgE22aCZ",
- "outputId": "c296203d-0fde-487e-a478-ad28af067cec"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Last updated: 2024-04-26 01:26:48.838163\n"
- ]
- }
- ],
- "source": [
- "# Quick timestamp\n",
- "import datetime\n",
- "print(f\"Last updated: {datetime.datetime.now()}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "nYCWwASwpPNl"
- },
- "source": [
- "## What we're going to cover\n",
- "\n",
- "In this project, we're going to be introduced to the power of deep learning and more specifically, transfer learning using TensorFlow/Keras.\n",
- "\n",
- "We'll go through each of these in the context of the [6 step machine learning framework](https://dev.mrdbourke.com/zero-to-mastery-ml/a-6-step-framework-for-approaching-machine-learning-projects/):\n",
- "\n",
- "1. **Problem defintion** - Use computer vision to classify photos of dogs into different dog breeds.\n",
- "2. **Data** - 20,000+ images of dogs from 120 different dog breeds from the [Stanford Dogs dataset](http://vision.stanford.edu/aditya86/ImageNetDogs/).\n",
- "3. **Evaluation** - We'd like to beat the original paper's results (22% mean accuracy across all classes, **tip:** A good way to practice your skills is to find some results online and try to beat them).\n",
- "4. **Features** - Because we're using deep learning, our model will learn the features on its own.\n",
- "5. **Modelling** - We're going to use a pretrained convolutional neural network (CNN) and transfer learning.\n",
- "6. **Experiments** - We'll try different amounts of data with the same model to see the effects on our results.\n",
- "\n",
- "> **Note:** It's okay not to know these exact steps ahead of time. When starting a new project, it's often the case you'll figure it out as you go. These steps are only filled out because I've had practice working on several machine learning projects. You'll pick up these ideas overtime.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "## Table of contents\n",
- "\n",
- "1. Getting Setup\n",
- "2. Getting Data (dog images and their breeds)\n",
- "3. Exploring the data (exploratory data analysis)\n",
- "4. Creating training and test splits\n",
- "5. Turning our datasets into TensorFlow Dataset(s)\n",
- "6. Creating a neural network with TensorFlow\n",
- "7. Model 0 - Train a model on 10% of the training data\n",
- "8. Putting it all together: create, compile, fit\n",
- "9. Model 1 - Train a model on 100% of the training data\n",
- "10. Make and evaluate predictions of the best model\n",
- "11. Save and load the best model\n",
- "12. Make predictions on custom images with the best model (bringing Dog Vision πΆποΈ to life!)\n",
- "13. Key takeaways\n",
- "14. Extensions & exercises"
- ],
- "metadata": {
- "id": "F3hGBuLWciOO"
- }
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "BLK0aCjmo8j6"
- },
- "source": [
- "## Where can can you get help?\n",
- "\n",
- "All of the materials for this course are [available on GitHub](https://github.com/mrdbourke/zero-to-mastery-ml/tree/master).\n",
- "\n",
- "If you run into trouble, you can ask a question on the course [GitHub Discussions page](https://github.com/mrdbourke/zero-to-mastery-ml/discussions) there too.\n",
- "\n",
- "You can also:\n",
- "* Search for questions online and end up at places such as Stack Overflow (a great resource of developer-focused Q&A).\n",
- "* Ask AI Assistants such as [ChatGPT](https://chat.openai.com/), [Gemini](https://gemini.google.com/) and [Claude](https://claude.ai/) for help with various coding problems and errors."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "h05v4NwY7EHj"
- },
- "source": [
- "## Quick definitions\n",
- "\n",
- "Let's start by breaking down some of the most important topics we're going to go through.\n",
- "\n",
- "### What is TensorFlow/Keras?\n",
- "\n",
- "[TensorFlow](https://www.tensorflow.org/) is an open source machine learning and deep learning framework originally developed by Google. Inside TensorFlow, you can also use [Keras](https://keras.io/) which is another very helpful machine learning framework known for its ease of use.\n",
- "\n",
- "### Why use TensorFlow?\n",
- "\n",
- "TensorFlow allows you to manipulate data and write deep learning algorithms using Python code.\n",
- "\n",
- "It also has several built-in capabilities to leverage accelerated computing hardware (e.g. GPUs, Graphics Processing Units and TPUs, Tensor Processing Units).\n",
- "\n",
- "Many of world's largest companies [power their machine learning workloads with TensorFlow](https://www.tensorflow.org/about/case-studies).\n",
- "\n",
- "### What is deep learning?\n",
- "\n",
- "[Deep learning](https://en.wikipedia.org/wiki/Deep_learning) is a form of machine learning where data passes through a series of progressive layers which all contribute to learning an overall representation of that data.\n",
- "\n",
- "Each layer performs a pre-defined operation.\n",
- "\n",
- "The series of progressive layers combine to form what's referred to as a [**neural network**](https://en.wikipedia.org/wiki/Artificial_neural_network).\n",
- "\n",
- "For example, a photo may be turned into numbers (e.g. red, green and blue pixel values) and those numbers are then manipulated mathematically through each progressive layer to learn patterns in the photo.\n",
- "\n",
- "The \"deep\" in deep learning comes from the number of layers used in the neural network.\n",
- "\n",
- "So when someone says deep learning or (artificial neural networks), they're typically referring to same thing.\n",
- "\n",
- "> **Note:** Artificial intelligence (AI), machine learning (ML) and deep learning are all broad terms. You can think of AI as the overall technology, machine learning as a type of AI, and deep learning as a type of machine learning. So if someone refers to AI, you can often assume they are often talking about machine learning or deep learning."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "UY9qNLKyvRu7"
- },
- "source": [
- "### What can deep learning be used for?\n",
- "\n",
- "Deep learning is such a powerful technique that new use cases are being discovered everyday.\n",
- "\n",
- "Most of the modern forms of artifical intelligence (AI) applications you see, are powered by deep learning.\n",
- "\n",
- "Two of the most useful types of AI are predictive and generative.\n",
- "\n",
- "Predictive AI learns the relationship between data and labels such as photos of dog and their breeds ([supervised learning](https://en.wikipedia.org/wiki/Supervised_learning)). So that when it sees a new photo of a dog, it can predict its breed based on what its learned.\n",
- "\n",
- "[Generative AI](https://en.wikipedia.org/wiki/Generative_artificial_intelligence) generates something new given an input such as creating new text given input text.\n",
- "\n",
- "Some examples of Predictive AI problems include:\n",
- "* Tesla's [self-driving cars use deep learning](https://www.tesla.com/AI) use object detection models to power their computer vision systems.\n",
- "* Apple's Photos app uses deep learning to [recognize faces in images](https://machinelearning.apple.com/research/recognizing-people-photos) and create Photo Memories.\n",
- "* Siri and Google Assistant use deep learning to transcribe speech and understand voice commands.\n",
- "* [Nutrify](https://nutrify.app) (an app my brother and I build) uses predictive AI to recognize food in images.\n",
- "* [Magika](https://google.github.io/magika/) uses deep learning to classify a file into what type it is (e.g. `.jpeg`, `.py`, `.txt`).\n",
- "* [Text classification models](https://huggingface.co/MoritzLaurer/deberta-v3-large-zeroshot-v2.0) such as DeBERTa use deep learning to classify text into different categories such as \"positive\" and \"negative\" or \"spam\" or \"not spam\".\n",
- "\n",
- "Some examples of Generative AI problems include:\n",
- "* [Stable Diffusion](https://huggingface.co/spaces/stabilityai/stable-diffusion) uses generative AI to generate images given a text prompt.\n",
- "* [ChatGPT](https://chat.openai.com) and other large language models (LLMs) such as [Llama](https://llama.meta.com), [Claude](https://claude.ai), [Gemini](https://gemini.google.com) and [Mistral](https://mistral.ai) use deep learning to process text and return a response.\n",
- "* [GitHub Copilot](https://github.com/features/copilot) uses generative AI to generate code snippets given surrounding context.\n",
- "\n",
- "All of these AI use cases are powered by deep learning.\n",
- "\n",
- "And more often than not, whenever you get started on a deep learning problem, you'll start with transfer learning.\n",
- "\n",
- "![\"\n",](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-example-ai-problems-visualized.png?raw=true\")
\n",
- "\n",
- "*Example of different every day problems where AI/machine learning gets used.*"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "T6jkt4X8vRu8"
- },
- "source": [
- "### What is transfer learning?\n",
- "\n",
- "[Transfer learning](https://en.wikipedia.org/wiki/Transfer_learning) is one of the most powerful and useful techniques in modern AI and machine learning.\n",
- "\n",
- "It involves taking what one model (or neural network) has learned in a similar domain and applying to your own.\n",
- "\n",
- "In our case, we're going to use transfer learning to take the patterns a neural network has learned from the 1 million+ images and over 1000 classes in [ImageNet](https://www.image-net.org/) (a gold standard computer vision benchmark) and apply them to our own problem of recognizing dog breeds.\n",
- "\n",
- "However, this concept can be applied to many different domains.\n",
- "\n",
- "You could take a large language model (LLM) that has been pre-trained on *most* of the text on the internet and learned very well the patterns in naturual language and customize it for your own specific chat use-case.\n",
- "\n",
- "The biggest benefit of transfer learning is that it often allows you to get outstanding results with less data and time.\n",
- "\n",
- "![This image is an infographic that explains the process of using pre-trained models from Hugging Face and Keras to apply to custom data. The top section asks, \"Does my problem exist on Hugging Face/Keras?\" indicating a starting point for a solution search. The left section shows the ImageNet database webpage, representing a large dataset, leading into the center where a network graph signifies a \"Pretrained Model.\" This model is then applied to custom data, illustrated by a diagram featuring various dog images being processed and resulting in checkmarks, implying successful identification or classification. At the bottom, it emphasizes that \"Keras applications make using a pre-trained model as simple as calling a Python class,\" with a specific code example given for EfficientNet V2. The overall message is that these tools can handle various data types, not just images, and simplify machine learning tasks.](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-a-transfer-learning-workflow.png?raw=true\")
\n",
- "\n",
- "*A transfer learning workflow. Many publicly available models have been pretrained on large datasets such as ImageNet (1 million+ images). These models can then be applied to similar tasks downstream. For example, we can take a model pretrained on ImageNet and apply it to our Dog Vision πΆποΈ problem. This same process can be repeated for many different styles of data and problem.*"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Mr8xFrmA0qeG"
- },
- "source": [
- "## 1. Getting setup\n",
- "\n",
- "This notebook is designed to run in [Google Colab](https://colab.research.google.com/), an online Jupyter Notebook that provides free access to GPUs (Graphics Processing Units, we'll hear more on these later).\n",
- "\n",
- "For a quick rundown on how to use Google Colab, see their [introductory guide](https://colab.research.google.com/notebooks/basic_features_overview.ipynb) (it's quite similar to a Jupyter Notebook with a few different options).\n",
- "\n",
- "Google Colab also comes with many data science and machine learning libraries pre-installed, including TensorFlow/Keras."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "4ArvTa1_vRu9"
- },
- "source": [
- "### Getting a GPU on Google Colab\n",
- "\n",
- "Before running any code, we'll make sure our Google Colab instance is connected to a GPU.\n",
- "\n",
- "You can do this via going to Runtime -> Change runtime type -> GPU (this may restart your existing runtime).\n",
- "\n",
- "Why use a GPU?\n",
- "\n",
- "Since neural networks perform a large amount of calculations behind the scenes (the main one being [matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication)), you need a computer chip that perform these calculations quickly, otherwise you'll be waiting all day for a model to train.\n",
- "\n",
- "And in short, GPUs are much faster at performing matrix multiplications than CPUs.\n",
- "\n",
- "Why this is the case is behind the scope of this project (you can search \"why are GPUs faster than CPUs for machine learning?\" for more).\n",
- "\n",
- "The main thing to remember is: generally, in deep learning, GPUs = faster than CPUs.\n",
- "\n",
- "> **Note:** A good experiment would be to run the neural networks we're going to build later on with and without a GPU and see the difference in their training times.\n",
- "\n",
- "Ok, enough talking, let's start by importing TensorFlow!\n",
- "\n",
- "We'll do so using the common abbreviation `tf`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 35
- },
- "id": "Ao399CMu9888",
- "outputId": "2763f1d1-a627-4975-9ddc-4f37b99caf3b"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "'2.15.0'"
- ],
- "application/vnd.google.colaboratory.intrinsic+json": {
- "type": "string"
- }
- },
- "metadata": {},
- "execution_count": 2
- }
- ],
- "source": [
- "import tensorflow as tf\n",
- "tf.__version__"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "l4tz_cCL2our"
- },
- "source": [
- "Nice!\n",
- "\n",
- "> **Note:** If you want to run TensorFlow locally, you can follow the [TensorFlow installation guide](https://www.tensorflow.org/install).\n",
- "\n",
- "Now let's check to see if TensorFlow has access to a GPU (this isn't 100% required to complete this project but will speed things up dramatically).\n",
- "\n",
- "We can do so with the method [`tf.config.list_physical_devices()`](https://www.tensorflow.org/api_docs/python/tf/config/list_physical_devices)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "7uNWd2rs2fZW",
- "outputId": "dc08719f-6d9a-4a19-fc4f-4ad3631d86f7"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "[INFO] TensorFlow has GPU available to use. Woohoo!! Computing will be sped up!\n",
- "[INFO] Accessible devices:\n",
- "[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\n"
- ]
- }
- ],
- "source": [
- "# Do we have access to a GPU?\n",
- "device_list = tf.config.list_physical_devices()\n",
- "if \"GPU\" in [device.device_type for device in device_list]:\n",
- " print(f\"[INFO] TensorFlow has GPU available to use. Woohoo!! Computing will be sped up!\")\n",
- " print(f\"[INFO] Accessible devices:\\n{device_list}\")\n",
- "else:\n",
- " print(f\"[INFO] TensorFlow does not have GPU available to use. Models may take a while to train.\")\n",
- " print(f\"[INFO] Accessible devices:\\n{device_list}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "PDvfRmUqx0C5"
- },
- "source": [
- "## 2. Getting Data\n",
- "\n",
- "All machine learning (and deep learning) projects start with data.\n",
- "\n",
- "If you have no data, you have no project.\n",
- "\n",
- "If you have no project, you have no cool models to show your friends or improve your business.\n",
- "\n",
- "Not to worry!\n",
- "\n",
- "There are several options and locations to get data for a deep learning project.\n",
- "\n",
- "| Resource | Description |\n",
- "| :----- | :----- |\n",
- "| [Kaggle Datasets](https://www.kaggle.com/datasets) | A collection of datasets across a wide range of topics. |\n",
- "| [TensorFlow Datasets](https://www.tensorflow.org/datasets) | A collection of ready-to-use machine learning datasets ready for use under the `tf.data.Datasets` API. You can see a list of [all available datasets](https://www.tensorflow.org/datasets/catalog/overview#all_datasets) in the TensorFlow documentation. |\n",
- "| [Hugging Face Datasets](https://huggingface.co/datasets) | A continually growing resource of datasets broken into several different kinds of topics. |\n",
- "| [Google Dataset Search](https://datasetsearch.research.google.com/) | A search engine by Google specifically focused on searching online datasets. |\n",
- "| Original sources | Datasets which are made available by researchers or companies with the release of a product or research paper (sources for these will vary, they could be a link on a website or a link to an application form). |\n",
- "| Custom datasets | These are datasets comprised of your own custom source of data. You may build these from scratch on your own or have access to them from an existing product or service. For example, your entire photos library could be your own custom dataset or your entire notes and documents folder or your company's customer order history. |\n",
- "\n",
- "In our case, the dataset we're going to use is called the Stanford Dogs dataset (or ImageNet dogs, as the images are dogs separated from ImageNet).\n",
- "\n",
- "Because the Stanford Dogs dataset has been around for a while (since 2011, which as of writing this in 2024 is like a lifetime in deep learning), it's available from several resources:\n",
- "\n",
- "* The [original project website](http://vision.stanford.edu/aditya86/ImageNetDogs/) via link download.\n",
- "* Inside [TensorFlow datasets under `stanford_dogs`](https://www.tensorflow.org/datasets/catalog/stanford_dogs).\n",
- "* On [Kaggle as a downloadable dataset](https://www.kaggle.com/datasets/jessicali9530/stanford-dogs-dataset).\n",
- "\n",
- "The point here is that when you're starting out with practicing deep learning projects, there's no shortage of datasets available.\n",
- "\n",
- "However, when you start wanting to work on your own projects or within a company environment, you'll likely start to work on custom datasets (datasets you build yourself or aren't available publicly online).\n",
- "\n",
- "The main difference between existing datasets and custom datasets is that existing datasets often come preformatted and ready to use.\n",
- "\n",
- "Where as custom datasets often require some preprocessing before they're ready to use within a machine learning project.\n",
- "\n",
- "To practice formatting a dataset for a machine learning problem, we're going to download the Stanford Dogs dataset from the original website.\n",
- "\n",
- "Before we do so, the following code is an example of how we'd get the Stanford Dogs dataset from [TensorFlow Datasets](https://www.tensorflow.org/datasets)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "id": "b_md5OPJ98nb"
- },
- "outputs": [],
- "source": [
- "# Download the dataset into train and test split using TensorFlow Datasets\n",
- "# import tensorflow_datasets as tfds\n",
- "# ds_train, ds_test = tfds.load('stanford_dogs', split=['train', 'test'])"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "1tHF13s8UbzK"
- },
- "source": [
- "### Download data directly from Stanford Dogs website\n",
- "\n",
- "Our overall project goal is to build a computer vision model which performs better than the original Stanford Dogs paper (average of 22% accuracy per class across 120 classes).\n",
- "\n",
- "To do so, we need some data.\n",
- "\n",
- "Let's download the original Stanford Dogs dataset from the project website.\n",
- "\n",
- "The data comes in three main files:\n",
- "\n",
- "1. [Images](http://vision.stanford.edu/aditya86/ImageNetDogs/images.tar) (757MB) - `images.tar`\n",
- "3. [Annotations](http://vision.stanford.edu/aditya86/ImageNetDogs/annotation.tar) (21MB) - `annotation.tar`\n",
- "3. [Lists](http://vision.stanford.edu/aditya86/ImageNetDogs/lists.tar) with train/test splits (0.5MB) - `lists.tar`\n",
- "\n",
- "Our goal is to get a file structure like this:\n",
- "\n",
- "![\"dog](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-dog-vision-file-structure.png?raw=true\")
\n",
- "\n",
- "> **Note:** If you're using Google Colab for this project, remember that any data uploaded to the Google Colab session gets deleted if the session disconnects. So to save us redownloading the data every time, we're going to download it once and save it to Google Drive.\n",
- ">\n",
- "> **Resource:** For a good guide on getting data in and out of Google Colab, see the [Google Colab `io.ipynb` tutorial](https://colab.research.google.com/notebooks/io.ipynb#scrollTo=RWSJpsyKqHjH).\n",
- "\n",
- "To make sure we don't have to keep redownloading the data every time we leave and come back to Google Colab, we're going to:\n",
- "\n",
- "1. Download the data if it doesn't already exist on Google Drive.\n",
- "2. Copy it to Google Drive (because Google Colab connects nicely with Google Drive) if it isn't already there.\n",
- "3. If the data already exists on Google Drive (we've been through steps 1 & 2), we'll import it instead.\n",
- "\n",
- "There are two main options to connect Google Colab instances to Google Drive:\n",
- "\n",
- "1. Click \"Mount Drive\" in \"Files\" menu on the left.\n",
- "2. Mount programmatically with `from google.colab import drive` -> `drive.mount('/content/drive')`.\n",
- "\n",
- "More specifically, we're going to follow the following steps:\n",
- "\n",
- "1. Mount Google Drive.\n",
- "2. Setup constants such as our base directory to save files to, the target files we'd like to download and target URL we'd like to download from.\n",
- "3. Setup our target local path to save to.\n",
- "4. Check if the target files all exist in Google Drive and if they do, copy them locally.\n",
- "5. If the target files don't exist in Google Drive, download them from the target URL with the [`!wget`](https://www.gnu.org/software/wget/manual/wget.html) command.\n",
- "6. Create a file on Google Drive to store the download files.\n",
- "7. Copy the downloaded files to Google Drive for use later if needed.\n",
- "\n",
- "A fair few steps, but nothing we can't handle!\n",
- "\n",
- "Plus, this is all good practice for dealing with and manipulating data, a very important skill in the machine learning engineers toolbox.\n",
- "\n",
- "> **Note:** The following data download section is designed to run in Google Colab. If you are running locally, feel free to modify the code to save to a local directory instead of Google Drive."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "ZHOkXNmeDgZq",
- "outputId": "6c4427ad-b162-443e-f078-13fe9125db74"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Mounted at /content/drive\n",
- "[INFO] Copying Dog Vision files from Google Drive to local directory...\n",
- "[INFO] Source dir: drive/MyDrive/tensorflow/dog_vision_data -> Target dir: dog_vision_data\n",
- "[INFO] Good to go!\n"
- ]
- }
- ],
- "source": [
- "from pathlib import Path\n",
- "from google.colab import drive\n",
- "\n",
- "# 1. Mount Google Drive (this will bring up a pop-up to sign-in/authenticate)\n",
- "# Note: This step is specifically for Google Colab, if you're working locally, you may need a different setup\n",
- "drive.mount(\"/content/drive\")\n",
- "\n",
- "# 2. Setup constants\n",
- "# Note: For constants like this, you'll often see them created as variables with all capitals\n",
- "TARGET_DRIVE_PATH = Path(\"drive/MyDrive/tensorflow/dog_vision_data\")\n",
- "TARGET_FILES = [\"images.tar\", \"annotation.tar\", \"lists.tar\"]\n",
- "TARGET_URL = \"http://vision.stanford.edu/aditya86/ImageNetDogs\"\n",
- "\n",
- "# 3. Setup local path\n",
- "local_dir = Path(\"dog_vision_data\")\n",
- "\n",
- "# 4. Check if the target files exist in Google Drive, if so, copy them to Google Colab\n",
- "if all((TARGET_DRIVE_PATH / file).is_file() for file in TARGET_FILES):\n",
- " print(f\"[INFO] Copying Dog Vision files from Google Drive to local directory...\")\n",
- " print(f\"[INFO] Source dir: {TARGET_DRIVE_PATH} -> Target dir: {local_dir}\")\n",
- " !cp -r {TARGET_DRIVE_PATH} .\n",
- " print(\"[INFO] Good to go!\")\n",
- "\n",
- "else:\n",
- " # 5. If the files don't exist in Google Drive, download them\n",
- " print(f\"[INFO] Target files not found in Google Drive.\")\n",
- " print(f\"[INFO] Downloading the target files... this shouldn't take too long...\")\n",
- " for file in TARGET_FILES:\n",
- " # wget is short for \"world wide web get\", as in \"get a file from the web\"\n",
- " # -nc or --no-clobber = don't download files that already exist locally\n",
- " # -P = save the target file to a specified prefix, in our case, local_dir\n",
- " !wget -nc {TARGET_URL}/{file} -P {local_dir} # the \"!\" means to execute the command on the command line rather than in Python\n",
- "\n",
- " print(f\"[INFO] Saving the target files to Google Drive, so they can be loaded later...\")\n",
- "\n",
- " # 6. Ensure target directory in Google Drive exists\n",
- " TARGET_DRIVE_PATH.mkdir(parents=True, exist_ok=True)\n",
- "\n",
- " # 7. Copy downloaded files to Google Drive (so we can use them later and not have to re-download them)\n",
- " !cp -r {local_dir}/* {TARGET_DRIVE_PATH}/"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "xJULPU7gbAgC"
- },
- "source": [
- "Data downloaded!\n",
- "\n",
- "Nice work! This may seem like a bit of work but it's an important step with any deep learning project. Getting data to work with.\n",
- "\n",
- "Now if we get the contents of `local_dir` (`dog_vision_data`), what do we get?\n",
- "\n",
- "We can first make sure it exists with [`Path.exists()`](https://docs.python.org/3/library/pathlib.html#pathlib.Path.exists) and then we can iterate through its contents with [`Path.iterdir()`](https://docs.python.org/3/library/pathlib.html#pathlib.Path.iterdir) and print out the [`.name`](https://docs.python.org/3/library/pathlib.html#pathlib.PurePath.name) attribute of each file."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "YRs1gwSLbMma",
- "outputId": "24578ea1-4c55-4f7c-918b-85c22175d8fd"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "dog_vision_data/\n",
- " lists.tar\n",
- " images.tar\n",
- " annotation.tar\n"
- ]
- }
- ],
- "source": [
- "if local_dir.exists():\n",
- " print(str(local_dir) + \"/\")\n",
- " for item in local_dir.iterdir():\n",
- " print(\" \", item.name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "_6Cqw4Qfb3Cq"
- },
- "source": [
- "Excellent! That's exactly the format we wanted.\n",
- "\n",
- "Now you might've noticed that each file ends in `.tar`.\n",
- "\n",
- "What's this?\n",
- "\n",
- "Searching \"what is .tar?\", I found:\n",
- "\n",
- "> *In computing, tar is a computer software utility for collecting many files into one archive file, often referred to as a tarball, for distribution or backup purposes.*\n",
- ">\n",
- "> Source: [Wikipedia tar page](https://en.wikipedia.org/wiki/Tar_(computing)).\n",
- "\n",
- "Exploring a bit more, I found that the `.tar` format is similar to `.zip`, however, `.zip` offers compression, where as `.tar` mostly combines many files into one.\n",
- "\n",
- "So how do we \"untar\" the files in `images.tar`, `annotation.tar` and `lists.tar`?\n",
- "\n",
- "We can use the [`!tar`](https://www.gnu.org/software/tar/manual/tar.html) command (or just `tar` from outside of a Jupyter Cell)!\n",
- "\n",
- "Doing this will expand all of the files within each of the `.tar` archives.\n",
- "\n",
- "We'll also use a couple of flags to help us out:\n",
- "* The `-x` flag tells `tar` to extract files from an archive.\n",
- "* The `-f` flag specifies that the following argument is the name of the archive file.\n",
- "* You can combine flags by putting them together `-xf`.\n",
- "\n",
- "Let's try it out!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "id": "SxChC5P7-5GP"
- },
- "outputs": [],
- "source": [
- "# Untar images, notes/tags:\n",
- "# -x = extract files from the zipped file\n",
- "# -v = verbose\n",
- "# -z = decompress files\n",
- "# -f = tell tar which file to deal with\n",
- "!tar -xf dog_vision_data/images.tar\n",
- "!tar -xf dog_vision_data/annotation.tar\n",
- "!tar -xf dog_vision_data/lists.tar"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "lDpvKKOsjCKU"
- },
- "source": [
- "What new files did we get?\n",
- "\n",
- "We can check in Google Colab by inspecting the \"Files\" tab on the left.\n",
- "\n",
- "Or with Python by using [`os.listdir(\".\")`](https://docs.python.org/3/library/os.html#os.listdir) where `\".\"` means \"the current directory\"."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "gzcNamgnjEbl",
- "outputId": "ef5fb1a1-2ec5-48c9-d6ed-eb1bfcca0348"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "['.config',\n",
- " 'dog_vision_data',\n",
- " 'file_list.mat',\n",
- " 'drive',\n",
- " 'train_list.mat',\n",
- " 'Images',\n",
- " 'Annotation',\n",
- " 'test_list.mat',\n",
- " 'sample_data']"
- ]
- },
- "metadata": {},
- "execution_count": 8
- }
- ],
- "source": [
- "import os\n",
- "\n",
- "os.listdir(\".\") # \".\" stands for \"here\" or \"current directory\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "JMfbxmM7jn-u"
- },
- "source": [
- "Ooooh!\n",
- "\n",
- "Looks like we've got some new files!\n",
- "\n",
- "Specifically:\n",
- "\n",
- "* `train_list.mat` - a list of all the training set images.\n",
- "* `test_list.mat` - a list of all the testing set images.\n",
- "* `Images/` - a folder containing all of the images of dogs.\n",
- "* `Annotation/` - a folder containing all of the annotations for each image.\n",
- "* `file_list.mat` - a list of all the files (training and test list combined).\n",
- "\n",
- "Our next step is to go through them and see what we've got.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "3fRbMSbBU67C"
- },
- "source": [
- "## 3. Exploring the data\n",
- "\n",
- "Once you've got a dataset, before building a model, it's wise to explore it for a bit to see what kind of data you're working with.\n",
- "\n",
- "Exploring a dataset can mean many things.\n",
- "\n",
- "But a few rules of thumb when exploring new data:\n",
- "* **View at least 100+ random samples for a \"vibe check\".** For example, if you have a large dataset of images, randomly sample 10 images at a time and view them. Or if you have a large dataset of texts, what do some of them say? The same with audio. It will often be impossible to view all samples in your dataset, but you can start to get a good idea of what's inside by randomly inspecting samples.\n",
- "* ***Visualize, viuslaize, visualize!*** This is the data explorer's motto. Use it often. As in, it's good to get statistics about your dataset but it's often even better to view 100s of samples with your own eyes (see the point above).\n",
- "* **Check the distributions and other various statistics.** How many samples are there? If you're dealing with classification, how many classes and labels per class are there? Which classes don't you understand? If you don't have labels, investigate [clustering methods](https://developers.google.com/machine-learning/clustering/overview) to put similar samples close together.\n",
- "\n",
- "As Abraham Lossfunction says...\n",
- "\n",
- "![\"text](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-abraham-loss-function.png?raw=true\")
\n",
- "\n",
- "*A play on words of Abraham Lincoln's [famous quote](https://www.brainyquote.com/quotes/abraham_lincoln_109275) on sharpening an axe before cutting down a tree in theme of machine learning. Source: [Daniel Bourke X/Twitter](https://twitter.com/mrdbourke/status/1456087631641473033).*"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "SBuYrnJZWecW"
- },
- "source": [
- "### Our target data format\n",
- "\n",
- "Since our goal is to build a computer vision model to classify dog breeds, we need a way to tell our model what breed of dog is in what image.\n",
- "\n",
- "A common data format for a classification problem is to have samples stored in folders named after their class name.\n",
- "\n",
- "For example:\n",
- "\n",
- "![\"example](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-image-folder-structure.png?raw=true\")
\n",
- "\n",
- "In the case of dog images, we'd put all of the images labelled \"chihuahua\" in a folder called `chihuahua/` (and so on for all the other classes and images).\n",
- "\n",
- "We could split these folders so that training images go in `train/chihuahua/` and testing images go in `test/chihuahua/`.\n",
- "\n",
- "This is what we'll be working towards creating.\n",
- "\n",
- "> **Note:** This structure of folder format doesn't just work for only images, it can work for text, audio and other kind of classification data too."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "3VxsBnbEWW8G"
- },
- "source": [
- "\n",
- "### Exploring the file lists\n",
- "\n",
- "How about we check out the `train_list.mat`, `test_list.mat` and `full_list.mat` files?\n",
- "\n",
- "Searching online, for \"what is a .mat file?\", I found that [it's a MATLAB file](https://www.mathworks.com/help/matlab/import_export/mat-file-versions.html). Before Python became the default language for machine learning and deep learning, many models and datasets were built in [MATLAB](https://www.mathworks.com/products/matlab.html).\n",
- "\n",
- "Then I searched, \"how to open a .mat file with Python?\" and found an [answer on Stack Overflow](https://stackoverflow.com/questions/874461/read-mat-files-in-python) saying I could use the [`scipy` library](https://scipy.org/) (a scientific computing library).\n",
- "\n",
- "The good news is, Google Colab comes with `scipy` preinstalled.\n",
- "\n",
- "We can use the [`scipy.io.loadmat()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.io.loadmat.html) method to open a `.mat` file."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "PmVRq8MTLM5s",
- "outputId": "3d160475-0e50-407b-8575-e2c1c281673c"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "({'__header__': b'MATLAB 5.0 MAT-file, Platform: GLNXA64, Created on: Sun Oct 9 08:36:13 2011',\n",
- " '__version__': '1.0',\n",
- " '__globals__': [],\n",
- " 'file_list': array([[array(['n02085620-Chihuahua/n02085620_5927.jpg'], dtype='| \n", - " | class_name | \n", - "image_count | \n", - "
|---|---|---|
| 116 | \n", - "n02085936-Maltese_dog | \n", - "252 | \n", - "
| 53 | \n", - "n02088094-Afghan_hound | \n", - "239 | \n", - "
| 111 | \n", - "n02092002-Scottish_deerhound | \n", - "232 | \n", - "
| 103 | \n", - "n02112018-Pomeranian | \n", - "219 | \n", - "
| 54 | \n", - "n02107683-Bernese_mountain_dog | \n", - "218 | \n", - "
| \n", - " | class_name | \n", - "image_count | \n", - "
|---|---|---|
| 116 | \n", - "maltese_dog | \n", - "252 | \n", - "
| 53 | \n", - "afghan_hound | \n", - "239 | \n", - "
| 111 | \n", - "scottish_deerhound | \n", - "232 | \n", - "
| 103 | \n", - "pomeranian | \n", - "219 | \n", - "
| 54 | \n", - "bernese_mountain_dog | \n", - "218 | \n", - "
| \n", - " | image_count | \n", - "
|---|---|
| count | \n", - "120.000000 | \n", - "
| mean | \n", - "171.500000 | \n", - "
| std | \n", - "23.220898 | \n", - "
| min | \n", - "148.000000 | \n", - "
| 25% | \n", - "152.750000 | \n", - "
| 50% | \n", - "159.500000 | \n", - "
| 75% | \n", - "186.250000 | \n", - "
| max | \n", - "252.000000 | \n", - "
| \n", - " | class_name | \n", - "image_count | \n", - "
|---|---|---|
| 33 | \n", - "labrador_retriever | \n", - "3 | \n", - "
| 23 | \n", - "welsh_springer_spaniel | \n", - "4 | \n", - "
| 61 | \n", - "great_dane | \n", - "4 | \n", - "
| 64 | \n", - "curly_coated_retriever | \n", - "4 | \n", - "
| 100 | \n", - "sussex_spaniel | \n", - "5 | \n", - "
![\"This](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-anatomy-of-a-neural-network.png?raw=true\")
\n",
- "\n",
- "*General anatomy of a neural network. Neural networks are almost infinitely customisable. The main premise is that data goes in one end, gets manipulated by many small functions in an attempt to learn patterns/weights which represent the data to produce useful outputs. Note that \"patterns\" is an arbitrary term, youβll often hear \"embedding\", \"weights\", \"feature representation\", \"representation\" all referring to similar things.*\n",
- "\n",
- "Where the input layer takes in the data, the middle layer(s) perform calculations on the data and (hopefully) learn patterns (also called weights/biases) to represent the data and the output layer performs a final transformation on the learned patterns to make them usable in human applications.\n",
- "\n",
- "What goes into the middle layer(s)?\n",
- "\n",
- "That's an excellent question.\n",
- "\n",
- "Because there are so many different options.\n",
- "\n",
- "But two of the most popular modern kinds of neural network are Convolutional Neural Networks (CNNs) and Transformers (the Transformer is the \"T\" in GPT, Generative Pretrained Transformer).\n",
- "\n",
- "| Architecture | Description | Example Layers | Problem Examples |\n",
- "|:----- |:-----|:-----|:-----|\n",
- "| [Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) | A combination of fully connected layers as well as attention-based layers. | [`tf.keras.layers.Attention`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention), [`tf.keras.layers.Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) | NLP, Machine Translation, Computer Vision |\n",
- "| [Convolutional Neural Network](https://en.wikipedia.org/wiki/Convolutional_neural_network) | A combination of fully connected layers as well as convolutional-based layers. | [`tf.keras.layers.Conv2D`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D), [`tf.keras.layers.Dense`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) | Computer Vision, Audio Processing |\n",
- "\n",
- "Because our problem is in the computer space, we're going to use a CNN.\n",
- "\n",
- "And instead of crafting our own CNN from scratch, we're going to take an existing CNN model and apply it to our own problem, harnessing the wonderful superpower of transfer learning.\n",
- "\n",
- "> **Note:** You can build and use working neural networks with TensorFlow without knowing the intricate details that's going on the behind the scenes (that's what we're focused on). For an idea of the mathematical operations that make neural networks work, I'd recommend going through [3Blue1Brown's YouTube series on Neural Networks](https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi).\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "wRIwqSrRUtqm"
- },
- "source": [
- "### The magic of transfer learning\n",
- "\n",
- "**Transfer learning** is the process of getting an existing working model and adjusting it to your own problem.\n",
- "\n",
- "This works particularly well for neural networks.\n",
- "\n",
- "The main benefit of transfer learning is being able to get better results in less time with less data.\n",
- "\n",
- "How?\n",
- "\n",
- "An existing model may have the following features:\n",
- "\n",
- "* Trained on lots of data (in the case of computer vision, existing models are often pretrained on ImageNet, a dataset of 1M+ images, this means they've already learned patterns across many different kinds of images).\n",
- "* Crafted by expert researchers (large universities and companies such as Google and Meta often open-source their best models for others to try and use).\n",
- "* Trained of lots of computing hardware (the larger the model and the larger the dataset, the more compute power you need, not everyone has access to 10s, 100s or 1000s of GPUs).\n",
- "* Proven to perform well on a given task through several studies (this means it has a good chance on performing well on your task if it's similar).\n",
- "\n",
- "You may be thinking, ok so, this all sounds incredible, where can I get pretrained models?\n",
- "\n",
- "And the good news is, there are plenty of places to find pretrained models!\n",
- "\n",
- "| Resource | Description |\n",
- "| :--- | :--- |\n",
- "| [`tf.keras.applications`](https://www.tensorflow.org/api_docs/python/tf/keras/applications) | A module built-in to TensorFlow and Keras with a series of pretrained models ready to use. |\n",
- "| [KerasNLP](https://keras.io/keras_nlp/) and [KerasCV](https://keras.io/keras_cv/) | Two dedicated libraries for NLP (natural language processing) and CV (computer vision) each of which includes many modality-specific APIs and is capable of running with TensorFlow, JAX or PyTorch. |\n",
- "| [Hugging Face Models Hub](https://huggingface.co/models) | A large collection of pretrained models on a wide range on tasks, from computer vision to natural language processing to audio processing. |\n",
- "| [Kaggle Models](https://www.kaggle.com/models) | A huge collection of different pretrained models for many different tasks. |\n",
- "\n",
- "![\"Four](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-places-to-find-pretrained-models.png?raw=true\")
\n",
- "\n",
- "*Different locations to find pretrained models. This list is consistantly expanding as machine learning becomes more and more open-source.*\n",
- "\n",
- "> **Note:** For most new machine learning problems, if you're looking to get good results quickly, you should generally look for a pretrained model similar to your problem and use transfer learning to adapt it to your own domain.\n",
- "\n",
- "Since we're focused on TensorFlow/Keras, we're going to be using a pretrained model from `tf.keras.applications`.\n",
- "\n",
- "More specifically, we're going to take the [`tf.keras.applications.efficientnet_v2.EfficientNetV2B0()`](https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet_v2/EfficientNetV2B0) model from the 2021 machine learning paper [*EfficientNetV2: Smaller Models and Faster Training*](https://arxiv.org/abs/2104.00298) from Google Research and apply it to our own problem.\n",
- "\n",
- "This model has been trained on [ImageNet1k](https://en.wikipedia.org/wiki/ImageNet) (1M+ images across 1000 different diverse classes, there is a version called ImageNet22k with 14M+ images across 22,000 categories) so it has a good baseline understanding of patterns in images across a wide domain.\n",
- "\n",
- "We'll see if we can adjust those patterns slightly to our dog images.\n",
- "\n",
- "Let's create an instance of it and call it `base_model` (I'll explain why next)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 57,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "S1tWQe3dHfeZ",
- "outputId": "8e1af304-b12a-4037-810e-c4f902f177cb"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/efficientnet_v2/efficientnetv2-b0.h5\n",
- "29403144/29403144 [==============================] - 0s 0us/step\n"
- ]
- }
- ],
- "source": [
- "# Create the input shape to our model\n",
- "INPUT_SHAPE = (*IMG_SIZE, 3)\n",
- "\n",
- "base_model = tf.keras.applications.efficientnet_v2.EfficientNetV2B0(\n",
- " include_top=True, # do want to include the top layer? (ImageNet has 1000 classes, so the top layer is formulated for this, we want to create our own top layer)\n",
- " include_preprocessing=True, # do we want the network to preprocess our data into the right format for us? (yes)\n",
- " weights=\"imagenet\", # do we want the network to come with pretrained weights? (yes)\n",
- " input_shape=INPUT_SHAPE # what is the input shape of our data we're going to pass to the network? (224, 224, 3) -> (height, width, colour_channels)\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "o68LDPUUbpJZ"
- },
- "source": [
- "Base model created!\n",
- "\n",
- "We can find out information about our base model by calling `base_model.summary()`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 58,
- "metadata": {
- "id": "3IQe-aO4aOGp",
- "tags": []
- },
- "outputs": [],
- "source": [
- "# Note: Uncomment to see full output\n",
- "# base_model.summary()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "6nBMeHrVa1-y"
- },
- "source": [
- "Truncated output of `base_model.summary()`:\n",
- "\n",
- "![\"Screenshot](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-effnetv2b0-model-summary.png?raw=true\")
\n",
- "\n",
- "Woah! Look at all those layers... this is what the \"deep\" in deep learning means! A *deep* number of layers.\n",
- "\n",
- "How about we count the number of layers?"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 59,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "eTJI8c-FxGMm",
- "outputId": "ed209d4c-0fff-4eb1-a91c-12ef5b02d32a"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Number of layers in base_model: 273\n"
- ]
- }
- ],
- "source": [
- "# Count the number of layers\n",
- "print(f\"Number of layers in base_model: {len(base_model.layers)}\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jpl3Va2-w-Wv"
- },
- "source": [
- "273 layers!\n",
- "\n",
- "Wow, there's a lot going on.\n",
- "\n",
- "Rather than step through each layer and explain what's happening in each layer, I'll leave that for the curious mind to research on their own.\n",
- "\n",
- "Just know that when starting out deep learning you don't *need* to know what's happening every layer in a model to be able to use a model.\n",
- "\n",
- "For now, let's pay attention to a few things:\n",
- "\n",
- "* The **input layer** (the first layer) input shape, this will tell us the shape of the data the model expects as input.\n",
- "* The **output layer** (the last layer) output shape, this will tell us the shape of the data the model will output.\n",
- "* The **number of parameters** of the model, these are \"learnable\" numbers (also called weights) that a model will use to derive patterns out of and represent the data. Generally, the more parameters a model has, the more learning capacity it has.\n",
- "* The **number of layers** a model has. Generally, the more layers a model has, the more learning capacity it has (each layer will learn progressively deeper patterns from the data). However, this caps out at a certain range.\n",
- "\n",
- "Let's step through each of these.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "uE1wOxjCMVAg"
- },
- "source": [
- "### Model input and output shapes\n",
- "\n",
- "One of the most important practical steps in using a deep learning model is input and output shapes.\n",
- "\n",
- "Two questions to ask:\n",
- "\n",
- "* What is the shape of my input data?\n",
- "* What is the ideal shape of my output data?\n",
- "\n",
- "We ask about shapes because in all deep learning models input and output data comes in the form of tensors.\n",
- "\n",
- "This goes for text, audio, images and more.\n",
- "\n",
- "The raw data gets converted to a numerical representation first before being passed to a model.\n",
- "\n",
- "In our case, our input data has the shape of `[(32, 224, 224, 3)]` or `[(batch_size, height, width, colour_channels)]`.\n",
- "\n",
- "And our ideal output shape will be `[(32, 120)]` or `[(batch_size, number_of_dog_classes)`.\n",
- "\n",
- "Your input and output shapes will differ depending on the problem and data you're working with.\n",
- "\n",
- "But as you get deeper into the world of machine learning (and deep learning), you'll find input and output shapes are one of the most common errors.\n",
- "\n",
- "We can check our model's input and output shapes with the `.input_shape` and `.output_shape` attributes.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 60,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "ekRXrqUnNwBD",
- "outputId": "c3848ecc-e33f-401c-df5b-9365c261ca6d"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "(None, 224, 224, 3)"
- ]
- },
- "metadata": {},
- "execution_count": 60
- }
- ],
- "source": [
- "# Check the input shape of our model\n",
- "base_model.input_shape"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "y2SUMMxiOBOo"
- },
- "source": [
- "Nice! Looks like our model's input shape is where we want it (remember `None` in this case is equivalent to a wild card dimension, meaning it could be any value, but we've set ours to `32`).\n",
- "\n",
- "This is because the model we chose, `tf.keras.applications.efficientnet_v2.EfficientNetV2B0`, has been trained on images the same size as our images.\n",
- "\n",
- "If our model had a different input shape, we'd have to make sure we processed our images to be the same shape.\n",
- "\n",
- "Now let's check the output shape."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 61,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "fEsmGs-ZNt4K",
- "outputId": "bbeabd1a-cbc8-4286-93ad-fc304883d222"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "(None, 1000)"
- ]
- },
- "metadata": {},
- "execution_count": 61
- }
- ],
- "source": [
- "# Check the model's output shape\n",
- "base_model.output_shape"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "iTLm2OaoOj67"
- },
- "source": [
- "Hmm, is this what we're after?\n",
- "\n",
- "Since we have 120 dog classes, we'd like an output shape of `(None, 120)`.\n",
- "\n",
- "Why is it by default `(None, 1000)`?\n",
- "\n",
- "This is because the model has been trained already on ImageNet, a dataset of 1,000,000+ images with 1000 classes (hence the `1000` in the output shape).\n",
- "\n",
- "How can we change this?\n",
- "\n",
- "Let's recreate a `base_model` instance, except this time we'll change the `classes` parameter to 120.\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 62,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 373
- },
- "id": "T5p4JqMirT8z",
- "outputId": "1dfa100c-67e6-47fb-f1f0-67fb475198a1"
- },
- "outputs": [
- {
- "output_type": "error",
- "ename": "ValueError",
- "evalue": "If using `weights` as `'imagenet'` with `include_top` as true, `classes` should be 1000Received: classes=120",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m![\"\n",](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-our-dog-vision-model.png?raw=true\")
\n",
- "\n",
- "*Example of how we can take a pretrained model and customize it to our own use case. This kind of transfer learning workflow is often referred to as a feature extracting workflow as the base layers are frozen (not changed during training) and only the top layers are trained. Note: In this image the EfficientNetB0 architecture is being demonstrated, however we're going to be using the EfficientNetV2B0 architecture which is slightly different. I've used the older architecture image from the research paper as a newer one wasn't available.*\n",
- "\n",
- "In other words, keep the patterns an existing model has learned on a similar problem (if they're good) to form a base representation of an input sample and then manipulate that base representation to suit our needs.\n",
- "\n",
- "Why do this?\n",
- "\n",
- "It's faster.\n",
- "\n",
- "The less trainable parameters, the faster your model training will be, the faster your experiments will be.\n",
- "\n",
- "But how will we know this works?\n",
- "\n",
- "We're going to run experiments to test it.\n",
- "\n",
- "Okay, so how do we freeze the parameters of our `base_model`?\n",
- "\n",
- "We can set its [`.trainable`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) attribute to `False`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 67,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "9uk1nb0XDIiQ",
- "outputId": "edaaea51-e855-4177-e11e-de62ff39a611"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "False"
- ]
- },
- "metadata": {},
- "execution_count": 67
- }
- ],
- "source": [
- "# Freeze the base model\n",
- "base_model.trainable = False\n",
- "base_model.trainable"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "eSBpX18gDK8b"
- },
- "source": [
- "`base_model` frozen!\n",
- "\n",
- "Now let's check the number of trainable and non-trainable parameters."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 68,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "B42oNpvFDQ8J",
- "outputId": "3c32841a-6fc4-4a15-d731-3c97f2628794"
- },
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "Model efficientnetv2-b0 parameter counts:\n",
- "Total parameters: 5919312.0\n",
- "Trainable parameters: 0.0\n",
- "Non-trainable parameters: 5919312\n"
- ]
- }
- ],
- "source": [
- "count_parameters(model=base_model, print_output=True)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "zw3zHNFhD6Nm"
- },
- "source": [
- "Beautiful!\n",
- "\n",
- "Looks like all of the parameters in our `base_model` are now non-trainable (frozen).\n",
- "\n",
- "This means they won't be updated during training."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "f-1WXphXBfok"
- },
- "source": [
- "### Passing data through our model\n",
- "\n",
- "We've spoken a couple of times how our `base_model` is a \"feature extractor\" or \"pattern extractor\".\n",
- "\n",
- "But what does this mean?\n",
- "\n",
- "It means that when a data sample goes through the `base_model`, its numbers get manipulated into a compressed set of features.\n",
- "\n",
- "In other words, the layers of the model will each perform a calculation on the sample eventually leading to an output tensor with patterns the model has deemed most important.\n",
- "\n",
- "This is often referred to a compressed feature space.\n",
- "\n",
- "That's one of the central ideas of deep learning.\n",
- "\n",
- "Take a large input (e.g. an image tensor of shape `[224, 224, 3]`) and compress it into a smaller output (e.g. a [**feature vector**](https://en.wikipedia.org/wiki/Feature_(machine_learning)#Feature_vectors) of shape `[1280]`) that captures a useful representation of the input.\n",
- "\n",
- "![\"A](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-feature-vector-extraction.png?raw=true\")
\n",
- "\n",
- "*Example of how a model can take an input piece of data and compress its representation into a feature vector with much lower dimensionality than the original data.*\n",
- "\n",
- "> **Note:** A feature vector is also referred to as an [**embedding**](https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture), a compressed representation of a data sample that makes it useful. The concept of embeddings is not limited to images either, the concept of embeddings stretches across all data types (text, images, video, audio + more).\n",
- "\n",
- "\n",
- "We can see this in action by passing a single image through our `base_model`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 69,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 321
- },
- "id": "EI9rHGtrGNWS",
- "outputId": "d6de4aa0-07d7-4d2a-b085-288f421fe524"
- },
- "outputs": [
- {
- "output_type": "error",
- "ename": "ValueError",
- "evalue": "Input 0 of layer \"efficientnetv2-b0\" is incompatible with the layer: expected shape=(None, 224, 224, 3), found shape=(224, 224, 3)",
- "traceback": [
- "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
- "\u001b[0;31mValueError\u001b[0m Traceback (most recent call last)",
- "\u001b[0;32m![\"A](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-how-a-neural-network-learns-on-dog-images-classification.png?raw=true\")
\n",
- "\n",
- "*Example of how a neural network learns (in brief). Note the cyclical nature of the learning. You can think of it as a big game of guess and check, where the guess (hopefully) get better over time.*\n",
- "\n",
- "I'll leave the intricacies of gradient descent and backpropagation to your own extra-curricula research.\n",
- "\n",
- "We're going to focus on using the tools TensorFlow has to offer to implement this process.\n",
- "\n",
- "As for optimizer functions, there are two main options to get started:\n",
- "\n",
- "| **Optimizer** | **Code** |\n",
- "| :----- | :----- |\n",
- "| [Stochastic Gradient Descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) (SGD) | [`tf.keras.optimizers.SGD()`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/SGD) or `\"sgd\"` for short. |\n",
- "| [Adam](https://arxiv.org/abs/1412.6980) | [`tf.keras.optimizers.Adam()`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam) or `\"adam\"` for short. |\n",
- "\n",
- "Why these two?\n",
- "\n",
- "Because they're the most often used in practice (you can see this via the number of machine learning papers referencing each one on [paperswithcode.com](https://paperswithcode.com/methods/category/optimization)).\n",
- "\n",
- "There are many more optimizers available in the [`tf.keras.optimizers`](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers) module too.\n",
- "\n",
- "The good thing about using a premade optimizer from `tf.keras.optimizers` is that they usually come with good starting settings.\n",
- "\n",
- "One the main ones being the `learning_rate` value.\n",
- "\n",
- "The `learning_rate` is one of the most important hyperparameters to set in a neural network training setup.\n",
- "\n",
- "It determines how much of a step change the optimizer will adjust your models weights every iteration.\n",
- "\n",
- "Too low and the model won't learn.\n",
- "\n",
- "Too high and the model will try to take too big of steps.\n",
- "\n",
- "By default, TensorFlow sets the learning rate of the Adam optimizer to 0.001 (`tf.keras.optimizers.Adam(learning_rate=0.001)`) which is a good setting for many problems to get started with.\n",
- "\n",
- "We can also set this default with the shortcut `optimizer=\"adam\"`.\n",
- "\n",
- "For more on finding the optimal learning rate, try searching for \"finding the optimal learning rate for neural networks\"."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 95,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "RZBFpQAYuFH9",
- "outputId": "7742de71-775c-4082-a6d6-7383fddbb4c2"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- "Generally with transfer learning you can get pretty good results quite quickly, however, you may want to look into training for longer (more epochs) as an experiment to see whether your model improves or not. What we've performed is a transfer learning technique called feature extraction, however, you may want to look further into fine-tuning (training the whole model to your own dataset) whole model and using callbacks (functions that take place during model training) such as Early Stopping to prevent the model from training so long its performance begins to degrade.
\n", - "As a potential extension, you may want to try practicing putting all of the steps we've been through so far together. As in, loading the data, creating the model, compiling the model, fitting the model and evaluating the model. That's what I've found is one of the best ways to learn ML problems, replicating a system end to end.\n", - "
\n", - "![\"\n",](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-stanford-dogs-dataset-results.png?raw=true\")
\n",
- "\n",
- "*Results from the original Stanford Dogs Dataset paper (2011). Let's see if the model we trained performs better than it.*\n",
- "\n",
- "How about we try and replicate the same plot with our own results?\n",
- "\n",
- "First, let's create a DataFrame with information about our test predictions and test samples.\n",
- "\n",
- "We'll start by getting the argmax of the test predictions as well as the test labels.\n",
- "\n",
- "Then we'll get the maximum prediction probabilities for each sample.\n",
- "\n",
- "And then we'll put it all into a DataFrame!"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 115,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 206
- },
- "id": "ANh2z09_N-Xm",
- "outputId": "7670f752-558b-429a-c17a-d6a4516e1e31"
- },
- "outputs": [
- {
- "output_type": "execute_result",
- "data": {
- "text/plain": [
- " test_pred_label test_pred_prob test_pred_class_name test_truth_label \\\n",
- "0 0 0.974350 affenpinscher 0 \n",
- "1 0 0.694450 affenpinscher 0 \n",
- "2 0 0.993829 affenpinscher 0 \n",
- "3 44 0.691742 flat_coated_retriever 0 \n",
- "4 0 0.989754 affenpinscher 0 \n",
- "\n",
- " test_truth_class_name correct \n",
- "0 affenpinscher True \n",
- "1 affenpinscher True \n",
- "2 affenpinscher True \n",
- "3 affenpinscher False \n",
- "4 affenpinscher True "
- ],
- "text/html": [
- "\n",
- " | \n", - " | test_pred_label | \n", - "test_pred_prob | \n", - "test_pred_class_name | \n", - "test_truth_label | \n", - "test_truth_class_name | \n", - "correct | \n", - "
|---|---|---|---|---|---|---|
| 0 | \n", - "0 | \n", - "0.974350 | \n", - "affenpinscher | \n", - "0 | \n", - "affenpinscher | \n", - "True | \n", - "
| 1 | \n", - "0 | \n", - "0.694450 | \n", - "affenpinscher | \n", - "0 | \n", - "affenpinscher | \n", - "True | \n", - "
| 2 | \n", - "0 | \n", - "0.993829 | \n", - "affenpinscher | \n", - "0 | \n", - "affenpinscher | \n", - "True | \n", - "
| 3 | \n", - "44 | \n", - "0.691742 | \n", - "flat_coated_retriever | \n", - "0 | \n", - "affenpinscher | \n", - "False | \n", - "
| 4 | \n", - "0 | \n", - "0.989754 | \n", - "affenpinscher | \n", - "0 | \n", - "affenpinscher | \n", - "True | \n", - "
| \n", - " | test_truth_class_name | \n", - "correct | \n", - "
|---|---|---|
| 10 | \n", - "bedlington_terrier | \n", - "1.000000 | \n", - "
| 62 | \n", - "keeshond | \n", - "1.000000 | \n", - "
| 30 | \n", - "chow | \n", - "0.989583 | \n", - "
| 92 | \n", - "saint_bernard | \n", - "0.985714 | \n", - "
| 2 | \n", - "african_hunting_dog | \n", - "0.985507 | \n", - "
| \n", - " | test_truth_class_name | \n", - "correct | \n", - "
|---|---|---|
| 104 | \n", - "staffordshire_bullterrier | \n", - "0.672727 | \n", - "
| 76 | \n", - "miniature_poodle | \n", - "0.654545 | \n", - "
| 90 | \n", - "rhodesian_ridgeback | \n", - "0.638889 | \n", - "
| 71 | \n", - "malamute | \n", - "0.615385 | \n", - "
| 101 | \n", - "siberian_husky | \n", - "0.271739 | \n", - "
| \n", - " | test_pred_label | \n", - "test_pred_prob | \n", - "test_pred_class_name | \n", - "test_truth_label | \n", - "test_truth_class_name | \n", - "correct | \n", - "
|---|---|---|---|---|---|---|
| 2727 | \n", - "75 | \n", - "0.997043 | \n", - "miniature_pinscher | \n", - "38 | \n", - "doberman | \n", - "False | \n", - "
| 5480 | \n", - "44 | \n", - "0.995325 | \n", - "flat_coated_retriever | \n", - "78 | \n", - "newfoundland | \n", - "False | \n", - "
| 6884 | \n", - "54 | \n", - "0.994142 | \n", - "groenendael | \n", - "95 | \n", - "schipperke | \n", - "False | \n", - "
| 4155 | \n", - "55 | \n", - "0.987126 | \n", - "ibizan_hound | \n", - "60 | \n", - "italian_greyhound | \n", - "False | \n", - "
| 1715 | \n", - "85 | \n", - "0.984834 | \n", - "pekinese | \n", - "22 | \n", - "brabancon_griffon | \n", - "False | \n", - "
![\"Two](\"https://github.com/mrdbourke/zero-to-mastery-ml/blob/master/images/unstructured-data-combining-models-for-deployment.png?raw=true\")
\n",
- "\n",
- "*Example of combining multiple machine learning models to create a workflow. One model for detecting food (Food Not Food) and another model for identifying what food is in the image (FoodVision, similar to Dog Vision). If an app is designed to take photos of food, taking photos of objects that aren't food and having them identified as food can be a poor customer experience. Source: [Nutrify](https://nutrify.app/).*\n",
- "\n",
- "These are some of the workflows you'll have to think about when you eventually deploy your own machine learning models.\n",
- "\n",
- "Machine learning models are often very powerful.\n",
- "\n",
- "But they aren't perfect.\n",
- "\n",
- "Implementing guidelines and checks around them is still a very active area of research."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "BXhsZtsUiF3J"
- },
- "source": [
- "## 13. Key Takeaways\n",
- "\n",
- "* **Data, data, data!** In any machine learning problem, getting a dataset and preparing it so that it is in a usable format will likely be the first and often most important step (hence why we spent so much time getting the data ready). It will also be an ongoing process, as although we've worked with thousands of dog images, our models could still be improved. And as we saw going from training with 10% of the data to 100% of the data, one of the best ways to improve a model is with more data. Explore your data early and often.\n",
- "* **When starting out, use transfer learning where possible.** For most new problems, you should generally look to see if a pretrained model exists and see if you can adapt it to your use case. Ask yourself: What format is my data in? What are my ideal inputs and outputs? Is there a pretrained model for my use case?\n",
- "* **TensorFlow and Keras provide building blocks** for neural networks which are powerful machine learning models capable of learning patterns in a wide range of data from text to audio to images and more.\n",
- "* **Experiment, experiment, experiment!** It's highly unlikely you'll ever get the best performing model on your first try. Machine learning is very experimental by nature. This includes experimenting on the data, the model, the training setup and the outputs (how does your model work in practice?). Always keep this front of mind in any machine learning project. Your results are never stationary and can often always be improved.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "jy8dvVj5oEXF"
- },
- "source": [
- "## Extensions & Exercises\n",
- "\n",
- "The following are a series of exercises and extensions which build on what we've covered throughout this module.\n",
- "\n",
- "I'd highly recommend going thorugh each one and spending time practicing what you've learned.\n",
- "\n",
- "This is where the real knowledge is built. Trying things out for yourself.\n",
- "\n",
- "1. Try a prediction with our trained model on your own images of dogs and see if the model is correct.\n",
- "2. Try training another model from `tf.keras.applications` (e.g. [ConvNeXt](https://www.tensorflow.org/api_docs/python/tf/keras/applications/convnext)) and see if it performs better than EfficientNetV2.\n",
- "3. Try training a model on your own images in different classes, for example, apple vs. banana vs. orange. You could download images from the internet and sort them into different folders and then load them how we've done in the data loading section. Or you could take photos of your own and build a model to differentiate between them.\n",
- "4. For more advanced model training, you may want to look into the concept of \"Callbacks\", these are functions which run during the model training. TensorFlow and Keras have a series of built-in callbacks which can be helpful for training. Have a read of the [`tf.keras.callbacks.Callback`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback) documentation and see which ones may be useful to you.\n",
- "5. We touched on the concept of overfitting when we trained our model. This is when a model performs far better on the training set than on the test set. The concept of trying to prevent overfitting is known as regularization. Spend 20-minutes researching \"ways to prevent overfitting\" and write a list of 2-3 techniques and how they might come into play with our model training. Tip: One of the most common regularization techniques in computer vision is [data augmentation](https://www.tensorflow.org/tutorials/images/data_augmentation) (also see the brief example below).\n",
- "6. One of the most important parts of machine learning is having good data. The next most important part is loading that data in a way that can used to train models as fast and efficiently as possible. For more on this, I'd highly recommend reading more about the [`tf.data` API](https://www.tensorflow.org/guide/data) (this API is TensorFlow focused, however, the concepts can be bridged to other dataloading needs) as well as reviewing the [`tf.data` best practices](https://www.tensorflow.org/guide/data_performance) (better performance with the `tf.data` API).\n",
- "7. Right now our model works well, however, we have to write code to interact with it. You could turn it into a small machine learning app using [Gradio](https://www.gradio.app/) so people can upload their own images of dogs and see what the model predicts. See the [example for image classification with TensorFlow and Keras](https://www.gradio.app/guides/image-classification-in-tensorflow) for an idea of what you could build. See an example of this below as well as a running [demo of Dog Vision on Hugging Face](https://huggingface.co/spaces/mrdbourke/dog_vision).\n",
- "\n",
- "In this project we've only really scratched the surface of what's possible with TensorFlow/Keras and deep learning.\n",
- "\n",
- "For a more comprehensive overview of TensorFlow/Keras, see the following:\n",
- "* [14-hour TensorFlow Tutorial on YouTube](https://youtu.be/tpCFfeUEGs8?si=HqWQ9BdLkV6YWgcF) (this is the first 14 hours of the ZTM TensorFlow course).\n",
- "* [Zero to Mastery TensorFlow for Deep Learning course](https://dbourke.link/ZTMTFcourse) (a 50+ hour course diving into many applications of TensorFlow and deep learning).\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "3K5fwoN4qE9n"
- },
- "source": [
- "### Extension example: data augmentation\n",
- "\n",
- "Data augmentation is a regularization technique to help prevent overfitting.\n",
- "\n",
- "It's designed to alter training images to artifically increase the diversity of the training dataset and hopefully help to generalize better to test images as well as real-life images.\n",
- "\n",
- "For example, we want our models to be able to identify the same breed of dog in an image regardless if the dog is facing left or right.\n",
- "\n",
- "So one simple data augmentation technique is to randomly flip the image horizontally so the model learns to recognize the same dog from different points of view.\n",
- "\n",
- "You can repeat this for many different types of image modifications such as rotation, zone, colour alterations and more.\n",
- "\n",
- "The following code is a brief example of how to incorporate a data augmentation layer into a model (note that in practice data augmentation is only applied during training time and not during testing/prediction time, this is set automatically within layers aimed at data augmentation).\n",
- "\n",
- "For more, see the [TensorFlow guide on data augmentation](https://www.tensorflow.org/tutorials/images/data_augmentation)."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 139,
- "metadata": {
- "id": "hWmXLLlZ6q-R"
- },
- "outputs": [],
- "source": [
- "from tensorflow.keras import layers\n",
- "\n",
- "# Note: Could functionize all of this\n",
- "\n",
- "# Setup hyperparameters\n",
- "img_size = 224\n",
- "num_classes = 120\n",
- "\n",
- "# Create data augmentation layer\n",
- "data_augmentation_layer = tf.keras.Sequential(\n",
- " [\n",
- " layers.RandomFlip(\"horizontal\"), # randomly flip image across horizontal axis\n",
- " layers.RandomRotation(factor=0.2), # randomly rotate image\n",
- " layers.RandomZoom(height_factor=0.2, width_factor=0.2) # randomly zoom into image\n",
- " # More augmentation can go here\n",
- " ],\n",
- " name=\"data_augmentation\"\n",
- ")\n",
- "\n",
- "# Setup base model\n",
- "base_model = tf.keras.applications.efficientnet_v2.EfficientNetV2B0(\n",
- " include_top=False,\n",
- " weights='imagenet',\n",
- " input_shape=(img_size, img_size, 3),\n",
- " include_preprocessing=True\n",
- ")\n",
- "\n",
- "# Freeze the base model\n",
- "base_model.trainable = False\n",
- "\n",
- "# Create new model\n",
- "inputs = tf.keras.Input(shape=(224, 224, 3))\n",
- "\n",
- "# Create data augmentation\n",
- "x = data_augmentation_layer(inputs)\n",
- "\n",
- "# Craft model\n",
- "x = base_model(x, training=False)\n",
- "x = tf.keras.layers.GlobalAveragePooling2D()(x)\n",
- "x = tf.keras.layers.Dropout(0.2)(x)\n",
- "outputs = tf.keras.layers.Dense(num_classes,\n",
- " name=\"output_layer\",\n",
- " activation=\"softmax\")(x) # Note: If you have \"softmax\" activation, use from_logits=False in loss function\n",
- "model_2 = tf.keras.Model(inputs, outputs, name=\"model_2\")\n",
- "\n",
- "# Uncomment for full model summary with augmentation layers\n",
- "# model_2.summary()"
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "### Extension Example: Gradio App Demo\n",
- "\n",
- "This is a modified version of the [Gradio Image Classification Tutorial with TensorFlow and Keras](https://www.gradio.app/guides/image-classification-in-tensorflow).\n",
- "\n",
- "You can see a [guide on Hugging Face](https://huggingface.co/blog/gradio-spaces) for how to host it on Hugging Face Spaces (a place where you can host and share your machine learning apps).\n",
- "\n",
- "First we'll [install Gradio](https://www.gradio.app/guides/quickstart)."
- ],
- "metadata": {
- "id": "xIGq7deUyUVG"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "!pip install -q gradio\n",
- "import gradio as gr"
- ],
- "metadata": {
- "id": "m_gmZq41x_o6",
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "outputId": "3bf4e266-4d7a-4d7e-b9a1-65dda258b34d"
- },
- "execution_count": 140,
- "outputs": [
- {
- "output_type": "stream",
- "name": "stdout",
- "text": [
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m12.2/12.2 MB\u001b[0m \u001b[31m34.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m91.9/91.9 kB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m314.4/314.4 kB\u001b[0m \u001b[31m33.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m141.1/141.1 kB\u001b[0m \u001b[31m18.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m8.8/8.8 MB\u001b[0m \u001b[31m91.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m47.2/47.2 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m60.8/60.8 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m129.9/129.9 kB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90mββββββββββββββββββββββββββββββββββββββββ\u001b[0m \u001b[32m71.9/71.9 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[?25h Building wheel for ffmpy (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
- "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
- "spacy 3.7.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.12.3 which is incompatible.\n",
- "weasel 0.3.4 requires typer<0.10.0,>=0.3.0, but you have typer 0.12.3 which is incompatible.\u001b[0m\u001b[31m\n",
- "\u001b[0m"
- ]
- }
- ]
- },
- {
- "cell_type": "markdown",
- "source": [
- "Then we'll download the saved model (the same model we trained during the Dog Vision notebook) along with the assosciated labels.\n",
- "\n",
- "I've stored my saved model as well as the Stanford Dogs class names on Hugging Face.\n",
- "\n",
- "You can see my files at [`huggingface.co/spaces/mrdbourke/dog_vision`](https://huggingface.co/spaces/mrdbourke/dog_vision/tree/main)."
- ],
- "metadata": {
- "id": "QGRZIiyjBdOk"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "import tensorflow as tf\n",
- "\n",
- "# Download saved model and labels from Hugging Face\n",
- "!wget -q https://huggingface.co/spaces/mrdbourke/dog_vision/resolve/main/dog_vision_model_demo.keras\n",
- "!wget -q https://huggingface.co/spaces/mrdbourke/dog_vision/resolve/main/stanford_dogs_class_names.txt\n",
- "\n",
- "# Load model\n",
- "model_save_path = \"dog_vision_model_demo.keras\"\n",
- "loaded_model_for_demo = tf.keras.models.load_model(model_save_path)\n",
- "\n",
- "# Load labels\n",
- "with open(\"stanford_dogs_class_names.txt\", \"r\") as f:\n",
- " class_names = [line.strip() for line in f.readlines()]"
- ],
- "metadata": {
- "id": "DgAO5H0g1bBP"
- },
- "execution_count": 141,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "The prediction function should take in an image and return a dictionary of classes and their prediction probabilities."
- ],
- "metadata": {
- "id": "URWk1NbGBOzW"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "# Create prediction function\n",
- "def pred_on_custom_image(image, # input image (preprocessed by Gradio's Image input to be numpy.array)\n",
- " model: tf.keras.Model = loaded_model_for_demo, # Trained TensorFlow model for prediction\n",
- " target_size: int = 224, # Desired size of the image for input to the model\n",
- " class_names: list = class_names): # List of class names\n",
- " \"\"\"\n",
- " Loads an image, preprocesses it, makes a prediction using a provided model,\n",
- " and returns a dictionary of prediction probabilities per class name.\n",
- "\n",
- " Args:\n",
- " image: Input image.\n",
- " model: Trained TensorFlow model for prediction.\n",
- " target_size (int, optional): Desired size of the image for input to the model. Defaults to 224.\n",
- " class_names (list, optional): List of class names for plotting. Defaults to None.\n",
- "\n",
- " Returns:\n",
- " Dict[str: float]: A dictionary of string class names and their respective prediction probability.\n",
- " \"\"\"\n",
- "\n",
- " # Note: gradio.inputs.Image handles opening the image\n",
- " # # Prepare and load image\n",
- " # custom_image = tf.keras.utils.load_img(\n",
- " # path=image_path,\n",
- " # color_mode=\"rgb\",\n",
- " # target_size=target_size,\n",
- " # )\n",
- "\n",
- " # Create resizing layer to resize the image\n",
- " resize = tf.keras.layers.Resizing(height=target_size,\n",
- " width=target_size)\n",
- "\n",
- " # Turn the image into a tensor and resize it\n",
- " custom_image_tensor = resize(tf.keras.utils.img_to_array(image))\n",
- "\n",
- " # Add a batch dimension to the target tensor (e.g. (224, 224, 3) -> (1, 224, 224, 3))\n",
- " custom_image_tensor = tf.expand_dims(custom_image_tensor, axis=0)\n",
- "\n",
- " # Make a prediction with the target model\n",
- " pred_probs = model.predict(custom_image_tensor)[0]\n",
- "\n",
- " # Predictions get returned as a dictionary of {label: pred_prob}\n",
- " pred_probs_dict = {class_names[i]: float(pred_probs[i]) for i in range(len(class_names))}\n",
- "\n",
- " return pred_probs_dict\n",
- "\n",
- "interface_title = \"Dog Vision πΆποΈ\"\n",
- "interface_description = \"Identify different dogs in images with deep learning. Model trained with TensorFlow/Keras.\"\n",
- "interface = gr.Interface(fn=pred_on_custom_image,\n",
- " inputs=gr.Image(),\n",
- " outputs=gr.Label(num_top_classes=3),\n",
- " examples=[\"dog-photo-1.jpeg\",\n",
- " \"dog-photo-2.jpeg\",\n",
- " \"dog-photo-3.jpeg\",\n",
- " \"dog-photo-4.jpeg\"],\n",
- " title=interface_title,\n",
- " description=interface_description)\n",
- "\n",
- "# Uncomment to launch the interface directly in a notebook\n",
- "# interface.launch(debug=True)"
- ],
- "metadata": {
- "id": "0S8-ePMz0ACQ"
- },
- "execution_count": 142,
- "outputs": []
- },
- {
- "cell_type": "markdown",
- "source": [
- "Save the following code to an `app.py` file for running on Hugging Face spaces.\n",
- "\n",
- "Finally, you can see the [running demo on Hugging Face](https://huggingface.co/spaces/mrdbourke/dog_vision).\n",
- "\n",
- "Try it out with your own images of dogs and see Dog Vision πΆποΈ come to life!\n"
- ],
- "metadata": {
- "id": "qtTRNQ9ZA5Wp"
- }
- },
- {
- "cell_type": "code",
- "source": [
- "from IPython.display import HTML\n",
- "\n",
- "# Embed the Hugging Face Space as an iframe\n",
- "html_string = \"\"\"\n",
- "\n",
- "\"\"\"\n",
- "\n",
- "display(HTML(html_string))"
- ],
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 871
- },
- "id": "cERok97lef1c",
- "outputId": "68382508-208a-404b-d08e-e862e6fe2aa7"
- },
- "execution_count": 143,
- "outputs": [
- {
- "output_type": "display_data",
- "data": {
- "text/plain": [
- "