3.Pytorch Basic of Neural Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "![](https://debuggercafe.com/wp-content/uploads/2019/11/lenet-5-e1574774376835.png)  \r\n",
    "### [LeNet-5 official Paper](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf)\r\n",
    "### Basic of Neural Network Notebook Covered\r\n",
    "1. Knowing the torch.nn package in pytorch\r\n",
    "2. Building a simple neural network architecture.\r\n",
    "3. Defining optimizers and loss functions in pytorch.\r\n",
    "4. How to backpropogate the gradients?"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### The torch.nn package\r\n",
    "\r\n",
    "- `torch.nn` package Module used as base class for all neural network.\r\n",
    "- `torch.nn` contains all the neural network layers such as Convolution and Linear and so on.\r\n",
    "- `torch.nn.functional` package define all the intermediate operations during the forward pass of the network, pooling operations, assiging the dropout and activation function etc."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Define Neural Network"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\r\n",
    "import torch.nn as nn\r\n",
    "import torch.nn.functional as F\r\n",
    "from torchsummary import summary"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "class Net(nn.Module):\r\n",
    "    def __init__(self):\r\n",
    "        super(Net, self).__init__()\r\n",
    "        self.conv1 = nn.Conv2d(1, 6, (3, 3)) # (input channel, output channel, kernel size)\r\n",
    "        self.conv2 = nn.Conv2d(6, 16, (3, 3)) # (input channel, output channel, kernel size)\r\n",
    "        \r\n",
    "        # In linear layer(output channels from Conv2d x width x height\r\n",
    "        self.fc1 = nn.Linear(16 * 6 * 6, 120) # (input channels, output channels)\r\n",
    "        self.fc2 = nn.Linear(120, 84) # (input channels, output channels)\r\n",
    "        self.fc3 = nn.Linear(84, 10) # (input channels, output channels)\r\n",
    "        \r\n",
    "    def forward(self, x):\r\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) # (input channel, output channel, kernel size)\r\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2)) # (input channel, output channel, kernel size)\r\n",
    "        x = x.view(x.size(0), -1) # Flatten the tensor\r\n",
    "        x = F.relu(self.fc1(x)) # (input channels, output channels)\r\n",
    "        x = F.relu(self.fc2(x)) # (input channels, output channels)\r\n",
    "        x = self.fc3(x) # (input channels, output channels)\r\n",
    "        return x # return the output\r\n",
    "    \r\n",
    "net = Net()\r\n",
    "print(net) \r\n",
    "print(str(summary(net, (1, 32, 32), depth=1)))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=576, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "C:\\Users\\SHIVA\\miniconda3\\envs\\pytorch19\\lib\\site-packages\\torch\\nn\\functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  ..\\c10/core/TensorImpl.h:1156.)\n",
      "  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "==========================================================================================\n",
      "Layer (type:depth-idx)                   Output Shape              Param #\n",
      "==========================================================================================\n",
      "├─Conv2d: 1-1                            [-1, 6, 30, 30]           60\n",
      "├─Conv2d: 1-2                            [-1, 16, 13, 13]          880\n",
      "├─Linear: 1-3                            [-1, 120]                 69,240\n",
      "├─Linear: 1-4                            [-1, 84]                  10,164\n",
      "├─Linear: 1-5                            [-1, 10]                  850\n",
      "==========================================================================================\n",
      "Total params: 81,194\n",
      "Trainable params: 81,194\n",
      "Non-trainable params: 0\n",
      "Total mult-adds (M): 0.27\n",
      "==========================================================================================\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.06\n",
      "Params size (MB): 0.31\n",
      "Estimated Total Size (MB): 0.38\n",
      "==========================================================================================\n",
      "==========================================================================================\n",
      "Layer (type:depth-idx)                   Output Shape              Param #\n",
      "==========================================================================================\n",
      "├─Conv2d: 1-1                            [-1, 6, 30, 30]           60\n",
      "├─Conv2d: 1-2                            [-1, 16, 13, 13]          880\n",
      "├─Linear: 1-3                            [-1, 120]                 69,240\n",
      "├─Linear: 1-4                            [-1, 84]                  10,164\n",
      "├─Linear: 1-5                            [-1, 10]                  850\n",
      "==========================================================================================\n",
      "Total params: 81,194\n",
      "Trainable params: 81,194\n",
      "Non-trainable params: 0\n",
      "Total mult-adds (M): 0.27\n",
      "==========================================================================================\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.06\n",
      "Params size (MB): 0.31\n",
      "Estimated Total Size (MB): 0.38\n",
      "==========================================================================================\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### `class Net(nn.Module)` Class Explained\r\n",
    "\r\n",
    "- Net class contains two important methods `__init__()` and `forward()`\r\n",
    "- `super(Net, self).__init__()` indicate that we inherit all the module property of `nn.Module()` and use all the methods and layers to define the neural network.\r\n",
    "- Then Begin to construct the neural network `nn.conv2d(input channels, output channels, kernel size)` [conv2d](https://pytorch.org/docs/stable/nn.html?highlight=conv2d#torch.nn.Conv2d).\r\n",
    "![Calculating the Height and Width](https://debuggercafe.com/wp-content/uploads/2019/11/Capture.png)  \r\n",
    "*Calculating the Height and Width*\r\n",
    "\r\n",
    "- where $Hin$=input height (let it be 32), $Win$=input width (let it be 32), $Hout$=output height, $Wout$=output width. And by default, padding is zero, dilaton is 1, stride is 1. Note that we have not given any values for padding, dilation, and stride, so the default values will be used.\r\n",
    "- Next `nn.Linear(input channels(features), output channels(features))` is used for Fully connected layers."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Define the Optimizer and Loss Function\r\n",
    "\r\n",
    "- OPTIMIZER - [Article](https://www.deeplearning.ai/ai-notes/optimization/)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "# Loss function and optimizer\r\n",
    "import torch.optim as optim # optimizer for the network\r\n",
    "loss_function = nn.MSELoss() # Mean Squared Error used for regression\r\n",
    "optimizer = optim.RMSprop(net.parameters(), lr = 0.001) # RMSprop is a variant of Adam based on adadelta"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Dummy Input and Backpropogation"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "torch.cuda.get_device_name(0)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "'NVIDIA GeForce RTX 3060 Laptop GPU'"
      ]
     },
     "metadata": {},
     "execution_count": 5
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Input(X)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "input = torch.randn(1, 1, 32, 32).to('cuda')\r\n",
    "out = net(input)\r\n",
    "print(out, out.size())"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[-0.0909, -0.0805,  0.0964, -0.0140, -0.0992, -0.0023, -0.0797,  0.2174,\n",
      "         -0.0558,  0.1773]], device='cuda:0', grad_fn=<AddmmBackward>) torch.Size([1, 10])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Label(y)"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "# Dummy variable\r\n",
    "labels = torch.rand(10).to('cuda')\r\n",
    "labels = labels.view(1,-1)\r\n",
    "print(labels, labels.shape)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[0.1771, 0.6598, 0.2284, 0.6371, 0.8484, 0.5922, 0.3972, 0.3106, 0.5181,\n",
      "         0.2712]], device='cuda:0') torch.Size([1, 10])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Loss Function and backward propagation"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "# define loss function\r\n",
    "loss = loss_function(out, labels)\r\n",
    "loss.backward()\r\n",
    "print(loss)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor(0.2887, device='cuda:0', grad_fn=<MseLossBackward>)\n"
     ]
    }
   ],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.8.0",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.0 64-bit ('pytorch19': conda)"
  },
  "interpreter": {
   "hash": "884a8f81666a19c0851426c83cd6eaa7b212468ad852fb3caa21591c98d4369f"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}