LeNet-Lab-Solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LeNet Lab Solution\n",
    "![LeNet Architecture](lenet.png)\n",
    "Source: Yan LeCun.\n",
    "Author: David Iglesias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test.shape\n",
    "y_test[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data\n",
    "\n",
    "Load the MNIST data, which comes pre-loaded with TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load pickled data\n",
    "import pickle\n",
    "\n",
    "# TODO: Fill this in based on where you saved the training and testing data\n",
    "training_file = 'traffic-signs-data/train.p'\n",
    "validation_file = 'traffic-signs-data/valid.p'\n",
    "testing_file = 'traffic-signs-data/test.p'\n",
    "\n",
    "with open(training_file, mode='rb') as f:\n",
    "    train = pickle.load(f)\n",
    "with open(validation_file, mode='rb') as f:\n",
    "    valid = pickle.load(f)\n",
    "with open(testing_file, mode='rb') as f:\n",
    "    test = pickle.load(f)\n",
    "    \n",
    "X_train, y_train = train['features'], train['labels']\n",
    "X_valid, y_valid = valid['features'], valid['labels']\n",
    "X_test, y_test = test['features'], test['labels']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create validation set from training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated image shape: (32, 32, 3)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, test_size=0.2, random_state=0)\n",
    "\n",
    "print(\"Updated image shape: {}\".format(X_train[0].shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Summary of the Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples = 27839\n",
      "Number of testing examples = 12630\n",
      "Image data shape = (32, 32, 3)\n",
      "Number of classes = 44\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# TODO: Number of training examples\n",
    "n_train = X_train.shape[0]\n",
    "\n",
    "# TODO: Number of validation examples\n",
    "n_validation = X_valid.shape[0]\n",
    "\n",
    "# TODO: Number of testing examples.\n",
    "n_test = X_test.shape[0]\n",
    "\n",
    "# TODO: What's the shape of an traffic sign image?\n",
    "image_shape = X_train.shape[1:]\n",
    "\n",
    "# TODO: How many unique classes/labels there are in the dataset.\n",
    "n_classes = 44\n",
    "\n",
    "print(\"Number of training examples =\", n_train)\n",
    "print(\"Number of testing examples =\", n_test)\n",
    "print(\"Image data shape =\", image_shape)\n",
    "print(\"Number of classes =\", n_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Data\n",
    "\n",
    "View a sample from the dataset.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 32, 3)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFsAAABZCAYAAABR/liSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEYtJREFUeJztnFuMXedVx39rX8595sycmfFcPPb4\n7thJnKRJUxRKRQVFCCQKDyD6gEBCCi+VQOKBiice+wC8IgVRiYdKCAnUixQoUZsChRLitG4d240d\nj8fX8die8VzPnMve++Nhrb3tOLHnxGN2ovgsabT37LP39+29vvWty3+t7xPnHH3Kh7yP+gUeJ+oz\nO0fqMztH6jM7R+ozO0fqMztH6jM7R9oWs0XkV0XkHRF5V0S+8qhe6pNK8rBBjYj4wFngC8AV4E3g\nS86504/u9T5ZFGzj2ReBd51zswAi8g/AF4H7MjvwfReGIXa/HfW3dNCdA7n3wfddeO/FDxYYa9/T\nYxD4dgwI7B1KxYoeC8aGJCJJEj33dNL7foDve9m7AcTtDgDLyyt0xaPZ3KDdbn3gW95N22H2TuDy\nXf9fAT5z700i8jLwMkAYBBzYMwMIoX285+kXdONIPyRy+DYCDv1wz/dJUn7a0ff8tAcie9Yhd92i\n54ViAYCR0YYex4ZoTOwE4IkDzwBwePeottm8zfrGmj5d0oEYGhqhNlDVdhPtc/m8fvY3vvltrvtl\nvvf9f30wp4y2w+wPGsn3iZhz7hXgFYByqeTiOIHAwxeT5DgGII7smIDYQKTSH8dxxsjQD9J2AVQS\ns17tGuBbG+l9zXVjomuStJoA+GMT2ua6tintJkm3C8ByaxmAqN1lZVH7XttQCT//7jUArqytU66F\nSI+qeDvMvgLsuuv/aeDagx9xgDInccY0m7VIOm0hMaZ5zlQBgmfTOh2AlImxDRaAkzsyLWb7A18l\nO4m1o/ZGTFtaAMyeOwfA6sK89p1ErDtl9oYNbjFO8Ey1rG/qDDp79QYANxevMbA6T7e7+eDPNtqO\nN/ImcFBE9opIAfhd4FvbaO8TTw8t2c65SES+DHwH8IGvOedOPfgZlcTQ9zNDlM7AwEvH3RHdo6BF\n5H1GMJVwz/MyfS82OyrFIkUzenGq401SB6o1KrUaAKumulxXj7VSkUKprPcV9VhMErxI1c6NhUUA\n2psr2ndYZd/hfZycu/qgz85oO2oE59yrwKvbaeNxom0x+8OTI3EJnufdcf18NWS+d0dynZgXYpeS\nOCK+R7IDP/VmIBQ9HxoaBuDFY88wUFXJPHnmZwDENluOHD1AbFLe6Tp7rg7AnvExaoMl7cA3F9Ul\nyOZtfa/mhrYVq14/dPhFfvkXXuD1//qfnr6+H67nSLlKtohHsVC0c5Uq36QSp+OeuDaSBiSp9HOX\nXy3mxaQeiweVgvrEB3ftAKDQWmS0fhiAZ/br8fz8HAA7JiYohNrW4tJNAAJPdTJJE8+pPk/QezYd\nVEtjADSbZ7X9gno4R49MM1ovEfhbxjPaT093PSISAd/zcC7JmBtzj7/t7vjUnhm8IAgy4xfFd4wm\ngOeFjAwMATBi/ayePsnAojKwMa0BzPWaDXKlSCM1ftb3RkvVRJR0ubWyrte6GiUmPoyberrebAMw\nWFa10ygL7WgV5+64nw+ivhrJkXI2kKBQhfee6BAgTvSYJA6LZbBgEfF8XOoOSmzX1IAFQZGxAQ3F\n5faq9rG2TlI6D8DAngEADs5MAlAredTMvUP0N1fUjjrtJklXjXM5VFXRTTaZu/g2AGcuXQJgZu/T\nAFQGK6y2lrJ33/Lbe7qrT4+E8tXZCL74JM6RIUupRKdht3iZ0fS9O7KQ6sVUd4P+Xyl5DJf13Lup\nQUfRL7K0rFgIFzW0PrTvUwAM1UcQsxcdT6W3ULSZEQS4tkp2vVKz9ytx4vgbAKx09LfKiFqH9TjB\njyULnLai3NUIAt3YIakvnXkeqbchuAz1MwPpgQGAGfpVMu9ktFKic/s6AIWOGrCRA0dpdZTZa/NL\nACyfvQjAZGOYpKRMTiPDpKXPhWEZv6Q9tNWVxq1v0l7Xf4KiGtlCTZ9fWlklbLUzEG0r6quRHClX\nyXY44jhCxOFlkaMeozgVXUeKiXhZYoHMv07MeoaFQQBGq6MkCyq1QVn94UOf+RU6HZX2M6/9CwAL\ncwrbTM2MU5lWsLLbUbQu6irWUQzreIG20erotXfP/4wbt9QfH6irizlaNwg4arG5sYpLEw5bUF+y\nc6TcDWToh/jEd+llO0rq58V4KdpnBjKK7qSrIrtWHFK3bdD3WbI01dCR/QBMHXuW0FMcY+3yHABn\nT58A4Oa1a+wYU+ntGnYdm72I2h26bdXxrY4+v9hpstLR9qdEdfbO2rjes3GNpDGUZZ22or5k50g5\nh+uCFxaI4i6xYdBpxjf0U4wEEkPVMncQyRLDZcMhhu3NWzeuUrDoZ+roIQAqI0OUfdXph546BsDC\n3BkALp8+RW1MMZTqlOYevaqG4wPlMuvrGrq3mhq2jw0PMDSobuDiir7zmXOqw/fvqTE0XMMPemNj\nzgYSYhLiJM4MonixvYj97xLCwKLLdJo7wSV6bUdFmThmTI/XlhmdPgDA5MEnAMUzYhu70Zl9AOze\no4DU6R+fZPbESQCemfwNAGoNZXbSWsPL/HcbwB0j7J2ZBuD42xqVnpl9E4DywH468SBR30B+/Chn\nAwmBQDt2SAqZWmSYyoYnkvl8aTI4imOwaG/nqGbES5aa2kA48JRGh9VhRePmLr1Da1ODmr2mMvZ+\n+rMAzF68yPJNTWPdmteShLFhxVm6m2ukQhqZwS5Xazx5RGfM/IK6kxevqEravafBYKlyB7fZgvqS\nnSPlG64L4Pk45K4KKEsiWNWRJy6rw0j1NHhUDasom5vWXNS6juLIBI29BwG4ta5u27df/TqLK/r7\nS5/7dT3ueRKAPc8+zekffA+Apcs/AWB4l0r22kaX2NPZsWNYUUKfmJKnmMvEmGIoF64q+nfqzGnK\nhUG6cW86O3dsxHmQuCSLury0MiqNFuM4SxoULA9YKhWY2qHgT9BW9SEGYE0ePkJtWv3e/z6phuuN\n4ydZNKD/dvgWAPt3HwVg6uiL3Dg3B8Ds25p5qQyrAazuniGOtKYkiPT5cqHIhkWto8Pqn++a0Puv\nXF/gP7tvsr6x0dO399VIjpS7n10uBFAukOJ3GWCaWO1HUKJk6Jp4+nrDA0McbOgU3pjVus1GQyX9\nyWPPUR9Ud3DN6jq6TfC7OiuWZ62mwyR9586D3DryHABX5/W3hXdULUxWR4kMrl00aa0kUCyrCttZ\n13dYGBq1d4Zri4vEUdTT9/clO0fKVbI9TygVCniJyxIDqVS0DJgX38MzY1kI9fUa1SrhuiJ0Xau3\nG9y3B4BifYK5S2oYm4Y7u/iumpOW4ho/Pal1ff4TESOH9wIwMav4yupV1d0jy1N4BXUtmx2dXX6h\njBOV8s2O6nPPAqrxxjS10gCXZ6/09v1b3SAiu0TkdRE5IyKnROSP7XpDRF4TkXN2HO6px8eYepHs\nCPhT59yPRGQAeEtEXgP+APiuc+6rtsTjK8CfPaghQfD9EN+PM28kjQfSrEwcxeA0g1LWOIaK38W7\nrV5IaHUnY0c1/L61ucG//fsPAZg9p5h1jMvqStLK09f+V0P0i6urfO4ldQN3Pv0SAMuXvwPAtXPn\nmB5Ul2+po9jI8tJtKlVt63qkx6io7uFQUGKiXOIHQbgVD4EemO2cmwfm7XxNRM6ghfBfBH7Rbvt7\n4PtswWyHFjquNNtZTtGZj9qxumhHghfotd1VNXxVr0VzZQGA8X3KqOmnjwCwSchzxzTCq0TqW19f\nmKNro1gItJzs+Sc1ynz++SeZGFfjWvS1uOfKpKqYhQsXqNU1ETG4bzcAG61VZq+oIb20rGqkUjY3\nVDp4hRJJipJtQR/KQIrIHuA54A1g3AYiHZAd93nmZRE5LiLHU4Y+rtSzgRSRGvBPwJ8451alx9G8\ne+VBfXDAOYG15jpRlKJ8711bI77DS1R/+LFO181bq8SxXpva/zwA5RGd7oPlAtPjeu6Wtaj99bd+\nSGLRp8VFHNij7trRA+OEZf3sQlWN4YFPPQVAc+ECy1dUFXmlyPoZolpUmRytqwprNlWlrbTaxAM1\nkkdZESUiIcrorzvn/tkuL4jIpP0+CdzoqcfHmLaUbFER/jvgjHPur+/66VvA7wNfteM3e2iLMPAJ\nPQ/x02UbdjTcz0mHalUlaGpQE6ytC9cpNCxUPqJhd8mqmhIvJvIs9K9r8BGMjrG7oSG1ZzUJ61Y8\n2SrEpDUnUlSxnzyms+Xm+TnOnvwxAP4NtRHOgSvbfQNqQ6KKBWR+xNrmSgY1bEW9qJGfB34POCki\nJ+zan6NM/kcR+UPgEvDbvXX5+FIv3sgPuM9KROCXPkxnIkIQFimXKrQtiZoYaF2rqE6uFjymG7oU\nrrih+HEcdzlkmPKOKQ2ZE1RiBYit/LjS0Jnwhc9/nh0Tuuzu1vwcAI16umCqTWzlwJEp9EJNg5xd\n+36O+fPvArA6b+XE4TCbTZ191WGdTeNTU9pfpUn7/Bo9mq/867ODQpEgCNhsqRvlWwqsUlUGD3k+\nw1Yu3VlRrKNR3cHkmLpilYJBrIbO0YmzdTm7J9Qh2jlUZ2hAkcDNXcqgzY7mFoO12ySWM7RKM4KW\nqq2RRpXxhqqizSVNLLiVG7StgqpT0sEZsOJkF/vs3rufgvn+W1EfG8mRck4eCIHvEztHYlM/sNJc\nChp8DA2PM2wYxMrCLQDCahHLC3P5uGLWsVMUL4k7RLY2phupavGiLgtxWrCp1yL7fyWJswAqMlwm\n2lSZC7ttog2NHJN0beP6TYYrOksi63NnQzGVUrnGYEWy1RRbUV+yc6Tcl3kEoY8fCOWySkPRXLh6\nVaWlXvNYu6CGMeqqJC3evMh/vPYNgAxT8WMzsJJk6bOs5i6J8dIUsh3S1WaJc5lRTleQpetndJ2m\n9hkba6LVDapVNSID4zr7BqycolQp4ElErxYydwMZFivUhxqULUGQWrexihpIaTbJEgvpPUlMsq64\nRwpYpUuuI5I7Oym41FeHNH2ZpJM3zeZ7Xlbqli7RS8xR9oMQQgUvo8SAsTjmuqmW5UVVa091Vf2U\nnK+AV4/12X01kiPlK9meR1CsMlgfpVZRxC01YGkVFBKQTFgSuK4uVhR1iPx0pwZb/2KuY9RpE0Wq\nUpptveaIs+L6dHWZu2ujgTTia5oa8cy1OzgzQ31Y++w4VRndzRYnTqlRvnBxDoCZc+8AcGD/DKVq\nkdj1K6I+dpS76yeBT+IiolilsNMx98typq1iwk3bYGX9tqajos4GxWHVpWmid8nqQqJmM5OsxTW9\nv9uJCUPV92WT2nbX2oqibPOYNVsZ5hXVXtR3FZia1uiwHqjh9hEuLuvOHhdPKSL4k9Nab1KqVRif\nHM0M7VbUl+wcKfdShiAs0Ipj1pY0SZtuulIfMoSv22V+RfHi1UVLhYmjYa5hqtq9dJujJL7j1sXp\nFkiAoYpDNcXE46b+1ok2EZN6ujq7krYGPB38LIPktXV2DQ/v4LMvfBqANVvucfGaJnhnlpbZOT52\nX+DoXsrd9SsUK7jEo9WyaC/d/cC3lVgFL9tlwbNjnEBsO++Edn9o+MYGZItUU3JJosv/ANL8oITW\nVgch3dHBHkwHqdsmwQbRpSsihPFRxVyeO6KpuAvf1fK12dkLHNl/gB61SF+N5EkPva/fQ3UmchMV\nxlu5dfrwNErv7znjnBvb6qZcmQ0gIsedcy/k2ulD0P/He/bVSI7UZ3aO9FEw+5WPoM+HoUf+nrnr\n7MeZ+mokR8qN2R/nvbYfUKn7FyJyVURO2N+vbaufPNTIx32vbavomry7Uhf4TeB3gHXn3F8+in7y\nkuxsr23nXAdI99r+WJBzbt459yM7XwPSSt1HSnkx+4P22n7kH/Mo6J5KXYAvi8hPReRr2y34z4vZ\nPe21/VHTvZW6wN8A+4Fn0Rr1v9pO+3kx+yH22s6XPqhS1zm34JyLnXMJ8LeoOnxoyovZH+u9tu9X\nqZuWRBv9FvD2dvrJBc9+mL22c6b7Vep+SUSeRVXeHPBH2+mkH0HmSP0IMkfqMztH6jM7R+ozO0fq\nMztH6jM7R+ozO0fqMztH+j9MHr0GuX6kzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11fc666a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "index = random.randint(0, len(X_train))\n",
    "image = X_train[index].squeeze()\n",
    "\n",
    "plt.figure(figsize=(1,1))\n",
    "plt.imshow(image)\n",
    "print(image.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle\n",
    "\n",
    "# Shuffle data\n",
    "X_train, y_train = shuffle(X_train, y_train)\n",
    "\n",
    "# Normalize data\n",
    "X_train = (X_train - 128) / 128\n",
    "X_valid = (X_valid - 128) / 128\n",
    "X_test = (X_test - 128) / 128\n",
    "\n",
    "# Augment data\n",
    "# TBD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup TensorFlow\n",
    "The `EPOCHS` and `BATCH_SIZE` values affect the training speed and model accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "EPOCHS = 10\n",
    "BATCH_SIZE = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SOLUTION: Implement LeNet-5\n",
    "\n",
    "### Input\n",
    "The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels.\n",
    "\n",
    "### Architecture\n",
    "**Layer 1: Convolutional.** The output shape should be 28x28x6.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 14x14x6.\n",
    "\n",
    "**Layer 2: Convolutional.** The output shape should be 10x10x16.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Pooling.** The output shape should be 5x5x16.\n",
    "\n",
    "**Flatten.** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using `tf.contrib.layers.flatten`, which is already imported for you.\n",
    "\n",
    "**Layer 3: Fully Connected.** This should have 120 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 4: Fully Connected.** This should have 84 outputs.\n",
    "\n",
    "**Activation.** Your choice of activation function.\n",
    "\n",
    "**Layer 5: Fully Connected (Logits).** This should have 10 outputs.\n",
    "\n",
    "### Output\n",
    "Return the result of the 2nd fully connected layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.contrib.layers import flatten\n",
    "\n",
    "def LeNet(x):    \n",
    "    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer\n",
    "    mu = 0\n",
    "    sigma = 0.1\n",
    "    \n",
    "    # SOLUTION: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.\n",
    "    conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 3, 6), mean = mu, stddev = sigma))\n",
    "    conv1_b = tf.Variable(tf.zeros(6))\n",
    "    conv1   = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b\n",
    "\n",
    "    # SOLUTION: Activation.\n",
    "    conv1 = tf.nn.relu(conv1)\n",
    "\n",
    "    # SOLUTION: Pooling. Input = 28x28x6. Output = 14x14x6.\n",
    "    conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "    # SOLUTION: Layer 2: Convolutional. Output = 10x10x16.\n",
    "    conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))\n",
    "    conv2_b = tf.Variable(tf.zeros(16))\n",
    "    conv2   = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    conv2 = tf.nn.relu(conv2)\n",
    "\n",
    "    # SOLUTION: Pooling. Input = 10x10x16. Output = 5x5x16.\n",
    "    conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')\n",
    "\n",
    "    # SOLUTION: Flatten. Input = 5x5x16. Output = 400.\n",
    "    fc0   = flatten(conv2)\n",
    "    \n",
    "    # SOLUTION: Layer 3: Fully Connected. Input = 400. Output = 120.\n",
    "    fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))\n",
    "    fc1_b = tf.Variable(tf.zeros(120))\n",
    "    fc1   = tf.matmul(fc0, fc1_W) + fc1_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    fc1    = tf.nn.relu(fc1)\n",
    "\n",
    "    # SOLUTION: Layer 4: Fully Connected. Input = 120. Output = 84.\n",
    "    fc2_W  = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))\n",
    "    fc2_b  = tf.Variable(tf.zeros(84))\n",
    "    fc2    = tf.matmul(fc1, fc2_W) + fc2_b\n",
    "    \n",
    "    # SOLUTION: Activation.\n",
    "    fc2    = tf.nn.relu(fc2)\n",
    "\n",
    "    # SOLUTION: Layer 5: Fully Connected. Input = 84. Output = 44.\n",
    "    fc3_W  = tf.Variable(tf.truncated_normal(shape=(84, 44), mean = mu, stddev = sigma))\n",
    "    fc3_b  = tf.Variable(tf.zeros(44))\n",
    "    logits = tf.matmul(fc2, fc3_W) + fc3_b\n",
    "    \n",
    "    return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Features and Labels\n",
    "Train LeNet to classify [MNIST](http://yann.lecun.com/exdb/mnist/) data.\n",
    "\n",
    "`x` is a placeholder for a batch of input images.\n",
    "`y` is a placeholder for a batch of output labels.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, (None, 32, 32, 3))\n",
    "y = tf.placeholder(tf.int32, (None))\n",
    "one_hot_y = tf.one_hot(y, 44)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Pipeline\n",
    "Create a training pipeline that uses the model to classify MNIST data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rate = 0.001\n",
    "\n",
    "logits = LeNet(x)\n",
    "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)\n",
    "loss_operation = tf.reduce_mean(cross_entropy)\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate = rate)\n",
    "training_operation = optimizer.minimize(loss_operation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "Evaluate how well the loss and accuracy of the model for a given dataset.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))\n",
    "accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "def evaluate(X_data, y_data):\n",
    "    num_examples = len(X_data)\n",
    "    total_accuracy = 0\n",
    "    sess = tf.get_default_session()\n",
    "    for offset in range(0, num_examples, BATCH_SIZE):\n",
    "        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]\n",
    "        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "        total_accuracy += (accuracy * len(batch_x))\n",
    "    return total_accuracy / num_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Model\n",
    "Run the training data through the training pipeline to train the model.\n",
    "\n",
    "Before each epoch, shuffle the training set.\n",
    "\n",
    "After each epoch, measure the loss and accuracy of the validation set.\n",
    "\n",
    "Save the model after training.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    num_examples = len(X_train)\n",
    "    \n",
    "    print(\"Training...\")\n",
    "    print()\n",
    "    for i in range(EPOCHS):\n",
    "        X_train, y_train = shuffle(X_train, y_train)\n",
    "        for offset in range(0, num_examples, BATCH_SIZE):\n",
    "            end = offset + BATCH_SIZE\n",
    "            batch_x, batch_y = X_train[offset:end], y_train[offset:end]\n",
    "            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y})\n",
    "            \n",
    "        validation_accuracy = evaluate(X_validation, y_validation)\n",
    "        print(\"EPOCH {} ...\".format(i+1))\n",
    "        print(\"Validation Accuracy = {:.3f}\".format(validation_accuracy))\n",
    "        print()\n",
    "        \n",
    "    saver.save(sess, './lenet')\n",
    "    print(\"Model saved\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model\n",
    "Once you are completely satisfied with your model, evaluate the performance of the model on the test set.\n",
    "\n",
    "Be sure to only do this once!\n",
    "\n",
    "If you were to measure the performance of your trained model on the test set, then improve your model, and then measure the performance of your model on the test set again, that would invalidate your test results. You wouldn't get a true measure of how well your model would perform against real data.\n",
    "\n",
    "You do not need to modify this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
    "\n",
    "    test_accuracy = evaluate(X_test, y_test)\n",
    "    print(\"Test Accuracy = {:.3f}\".format(test_accuracy))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}