OpenCV IV - Object Detection and Segmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Object Detection\n",
    "## 1.1) Haar Cascade\n",
    "\n",
    "The Haar feature-based cascade classifier is an effective object detection method proposed by Viola and Jones. It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. [1]\n",
    "\n",
    "Initially, the algorithm needs a lot of positive images and negative images to train the classifier. Then we need to extract features from it. For this, Haar features shown in the below image are used. They are just like our convolutional kernel. Each feature is a single value obtained by subtracting sum of pixels under the white rectangle from sum of pixels under the black rectangle. [1]\n",
    "\n",
    "<center>\n",
    "    <figure style=\"margin-top:20px\">\n",
    "        <img src=\"https://docs.opencv.org/2.4/_images/haarfeatures.png\">\n",
    "        <figcaption style=\"margin-top:20px\">Figura 1 - Kind of features [2].</figcaption>\n",
    "    </figure>\n",
    "</center>\n",
    "\n",
    "Instead of applying all features on a window, the features are grouped into different stages of classifiers and applied one-by-one. (Normally the first few stages will contain very many fewer features). If a window fails the first stage, discard it. We don't consider the remaining features on it. If it passes, apply the second stage of features and continue the process.\n",
    "\n",
    "Now, let's discover how to use this classifier with OpenCV. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing OpenCV library\n",
    "import cv2\n",
    "\n",
    "# Importing library to show our images\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# Importing library to manipulate our images\n",
    "import numpy as np\n",
    "\n",
    "from math import sqrt\n",
    "\n",
    "# Loading our image\n",
    "car = cv2.imread('images/car.jpg')\n",
    "car_gray = cv2.cvtColor(car, cv2.COLOR_BGR2GRAY)\n",
    "car = cv2.cvtColor(car, cv2.COLOR_BGR2RGB)\n",
    "\n",
    "# Showing our image\n",
    "plt.figure()\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(car)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initially, we need to load a classifier:\n",
    "\n",
    "<center>\n",
    "    <strong>cv2.CascadeClassifier([filename]) → Classifier Object </strong>\n",
    "</center>\n",
    "\n",
    "where:\n",
    "\n",
    "<ul>\n",
    "    <li><i>filename</i> – Name of the file from which the classifier is loaded.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading our classifier\n",
    "plates_classifier = cv2.CascadeClassifier('classifiers/plates.xml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we should use the following function to try to classify the object: \n",
    "\n",
    "<center>\n",
    "    <strong>*name_of_classifier*.detectMultiScale(image[, scaleFactor[, minNeighbors[, flags[, minSize[, maxSize]]]]]) → objects </strong>\n",
    "</center>\n",
    "\n",
    "where:\n",
    "\n",
    "<ul>\n",
    "    <li><i>image</i> – Matrix of the type CV_8U containing an image where objects are detected..</li>\n",
    "    <li><i>scaleFactor</i> – Parameter specifying how much the image size is reduced at each image scale.</li>\n",
    "    <li><i>minNeighbors</i> – Parameter specifying how many neighbors each candidate rectangle should have to retain it.</li>\n",
    "    <li><i>flags</i> – Parameter with the same meaning for an old cascade as in the function cvHaarDetectObjects. It is not used for a new cascade.</li>\n",
    "    <li><i>minSize</i> – Minimum possible object size. Objects smaller than that are ignored.</li>\n",
    "    <li><i>maxSize</i> – Maximum possible object size. Objects larger than that are ignored.</li>\n",
    "    <li><i>objects</i> – Vector of rectangles where each rectangle contains the detected object.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "min_width = car.shape[1] * 0.045\n",
    "\n",
    "# Detecting out plates\n",
    "plates = plates_classifier.detectMultiScale(car_gray, 1.1, 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we only need to isolate the rectangles which contain the objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drawing rects around detected plates\n",
    "car_cp = car.copy()\n",
    "\n",
    "for (x,y,w,h) in plates:\n",
    "    cv2.rectangle(car_cp,(x,y),(x+w,y+h),(255,0,0),2)\n",
    "\n",
    "dpi = 100\n",
    "height, width = car.shape[:2]\n",
    "\n",
    "figsize = width / dpi, height / dpi\n",
    "\n",
    "# Create a figure of the right size with one axes that takes up the full figure\n",
    "fig = plt.figure(figsize=figsize)\n",
    "plt.axis('off')\n",
    "plt.imshow(car_cp)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, as we already know our ```plates```, we will try to segment the characters of the plates. Now, we have to crop the plates from original image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the list to store our plates\n",
    "fig_plates = []\n",
    "fig_plates_ngray = []\n",
    "\n",
    "# Adding plates to the list\n",
    "for (x,y,w,h) in plates:\n",
    "    fig_plates.append( car_gray[y : y + h, x : x + w] )\n",
    "    fig_plates_ngray.append( car[y : y + h, x : x + w] )\n",
    "\n",
    "# Knowing the plates\n",
    "for p in fig_plates:\n",
    "    plt.figure()\n",
    "    plt.axis(\"off\")\n",
    "    plt.imshow(p, cmap='gray')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, let's to *binarize* the plates using **adaptative thresholding**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "binary_plates = []\n",
    "\n",
    "for i in range(0, len(fig_plates)):\n",
    "    binary_plates.append(cv2.adaptiveThreshold(fig_plates[i], 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 17, 5))\n",
    "\n",
    "for p in binary_plates:\n",
    "    plt.figure()\n",
    "    plt.axis(\"off\")\n",
    "    plt.imshow(p, cmap='gray')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Segmentation\n",
    "## 2.1) Connected components\n",
    "\n",
    "This is the moment for we segment the letters of the plates, and this operation is possible by the use of connected components analysis. It is an algorithmic application of graph theory, where subsets of connected components are uniquely labeled based on a given heuristic and is used to detect connected regions in binary digital images. [3]\n",
    "\n",
    "With OpenCV, we can use the following function:\n",
    "\n",
    "<center>\n",
    "    <strong>cv2.connectedComponentsWithStats(image, connectivity, type) </strong>\n",
    "</center>\n",
    "\n",
    "where:\n",
    "\n",
    "<ul>\n",
    "    <li><i>image</i> – The 8-bit single-channel image to be labeled.</li>\n",
    "    <li><i>connectivity</i> – 8 or 4 for 8-way or 4-way connectivity respectively.</li>\n",
    "    <li><i>type</i> – Output image label type. Currently CV_32S and CV_16U are supported.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Auxiliary function\n",
    "def euclidean(p, q):\n",
    "    if len(p) != len (q):\n",
    "        return -1\n",
    "    \n",
    "    local_sum = 0\n",
    "    for i in range(0, len(p)):\n",
    "        local_sum += pow(q[i] - p[i], 2)\n",
    "    \n",
    "    return sqrt (local_sum)\n",
    "\n",
    "# Trying to segment our plates\n",
    "for p in range(0, (len(binary_plates) - 1)):\n",
    "    output = cv2.connectedComponentsWithStats(binary_plates[p], 8, cv2.CV_32S)\n",
    "    num_labels = output[0]\n",
    "    labels = output[1]\n",
    "    stats = output[2]\n",
    "    \n",
    "    if num_labels < 7:\n",
    "        continue\n",
    "        \n",
    "    # Filtering by area\n",
    "    labels_area = []\n",
    "    \n",
    "    for label in range(0, num_labels - 1):\n",
    "        labels_area.append( (stats[label, cv2.CC_STAT_AREA], label) )\n",
    "    \n",
    "    labels_area = sorted(labels_area)\n",
    "    \n",
    "    # Choosing the 10 biggest\n",
    "    if len(labels_area) < 10:\n",
    "        labels_area = [label[1] for label in labels_area]\n",
    "    else:\n",
    "        labels_area = [label[1] for label in labels_area[len(labels_area) - 10 : len(labels_area)]]\n",
    "    \n",
    "    # Filtering by mean of width    \n",
    "    labels_width = []\n",
    "    mean_width = 0\n",
    "    \n",
    "    for label in labels_area:\n",
    "        mean_width += stats[label, cv2.CC_STAT_WIDTH]\n",
    "        labels_width.append( (stats[label, cv2.CC_STAT_WIDTH], label) )\n",
    "    \n",
    "    mean_width = mean_width / len(labels_area)\n",
    "    \n",
    "    for label in range(0, (len(labels_width) - 1)):\n",
    "        labels_width[label] = (euclidean([mean_width], [labels_width[label][0]]), labels_width[label][1])\n",
    "    \n",
    "    labels_width = sorted(labels_width)\n",
    "    final_labels = [label[1] for label in labels_width[:7]]\n",
    "    \n",
    "    # Drawing the result\n",
    "    result = fig_plates_ngray[p].copy()\n",
    "    \n",
    "    for label in final_labels:\n",
    "        width = stats[label, cv2.CC_STAT_WIDTH]\n",
    "        height = stats[label, cv2.CC_STAT_HEIGHT]\n",
    "        x = stats[label, cv2.CC_STAT_LEFT]\n",
    "        y = stats[label, cv2.CC_STAT_TOP]\n",
    "        cv2.rectangle(result,(x,y),(x+width,y+height),(255,0,0),1)\n",
    "    \n",
    "    plt.figure()\n",
    "    plt.axis(\"off\")\n",
    "    plt.imshow(result)\n",
    "    plt.show()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "[1] OpenCV: Face Detection using Haar Cascades. 2018. OpenCV: Face Detection using Haar Cascades. [ONLINE] Available at: https://docs.opencv.org/3.4.1/d7/d8b/tutorial_py_face_detection.html. [Accessed 04 May 2018].\n",
    "\n",
    "[2] Cascade Classification — OpenCV 2.4.13.6 documentation. 2018. Cascade Classification — OpenCV 2.4.13.6 documentation. [ONLINE] Available at: https://docs.opencv.org/2.4/modules/objdetect/doc/cascade_classification.html. [Accessed 04 May 2018].\n",
    "\n",
    "[3] Connected-component labeling - Wikipedia. 2018. Connected-component labeling - Wikipedia. [ONLINE] Available at: https://en.wikipedia.org/wiki/Connected-component_labeling. [Accessed 04 May 2018]."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}