CrowdFlower_Aanchan.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:b0335c8758238619ac23f5cb1b00af29536d5d7b2bbefc20dfa9e2e7a6fd5abe"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {
      "slideshow": {
       "slide_type": "slide"
      }
     },
     "source": [
      "Kaggle Crowd Flower Competition \n",
      "\n",
      "\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Useful links\n",
      "[Link](https://www.kaggle.com/c/crowdflower-search-relevance) to the competition \n",
      "\n",
      "[Link](https://www.kaggle.com/c/crowdflower-search-relevance/data) to the data \n",
      "\n",
      "[Link](https://github.com/ChenglongChen/Kaggle_CrowdFlower) to first place solution\n",
      "\n",
      "[Link](https://github.com/geffy/kaggle-crowdflower) to second place solution\n",
      "\n",
      "[Armaan's NLP tutorial](https://github.com/rmanak/nlp_tutorials) \n",
      "\n",
      "[Link](https://www.kaggle.com/abhishek/crowdflower-search-relevance/beating-the-benchmark) to a kind person's starter code\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#The task\n",
      "\n",
      "Given a query like **projector** and a search result like **ViewSonic Pro8200 DLP Multimedia Projector** humans rated the relevance to be 4 on a scale of 1-4\n",
      "\n",
      "Given this training set, can a machine be taught to do the same.\n",
      "\n",
      "    \n",
      " "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#First place solution overview\n",
      "<img src=\"FlowChart.jpg\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Second place solution overview\n",
      "<img src=\"second.jpg\">"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#import the require modules\n",
      "import numpy as np\n",
      "import pandas as pd\n",
      "import nltk\n",
      "import re\n",
      "from sklearn.feature_extraction import text\n",
      "from sklearn.svm import SVC\n",
      "from sklearn.pipeline import Pipeline\n",
      "import scipy.sparse as sp\n",
      "from sklearn.decomposition import TruncatedSVD\n",
      "from sklearn.preprocessing import StandardScaler\n",
      "from bs4 import BeautifulSoup"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#The data\n",
      "1. Note : This is a very small data set! Beware of overfitting!\n",
      "2. Also text data is very messy. The most messy is the product_description field (Use beautiful soup to parse out HTML tags).\n",
      "3. The data has spelling errors!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#load data frame train and test\n",
      "df_train = pd.read_csv(\"data/train.csv\").fillna(\"\")\n",
      "df_test = pd.read_csv(\"data/test.csv\").fillna(\"\")"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>id</th>\n",
        "      <th>query</th>\n",
        "      <th>product_title</th>\n",
        "      <th>product_description</th>\n",
        "      <th>median_relevance</th>\n",
        "      <th>relevance_variance</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 1</td>\n",
        "      <td> bridal shower decorations</td>\n",
        "      <td>       Accent Pillow with Heart Design - Red/Black</td>\n",
        "      <td> Red satin accent pillow embroidered with a hea...</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 2</td>\n",
        "      <td>      led christmas lights</td>\n",
        "      <td> Set of 10 Battery Operated Multi LED Train Chr...</td>\n",
        "      <td> Set of 10 Battery Operated Train Christmas Lig...</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0.000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4</td>\n",
        "      <td>                 projector</td>\n",
        "      <td>        ViewSonic Pro8200 DLP Multimedia Projector</td>\n",
        "      <td>                                                  </td>\n",
        "      <td> 4</td>\n",
        "      <td> 0.471</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 5</td>\n",
        "      <td>                 wine rack</td>\n",
        "      <td> Concept Housewares WR-44526 Solid-Wood Ceiling...</td>\n",
        "      <td> Like a silent and sturdy tree, the Southern En...</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0.000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 7</td>\n",
        "      <td>                light bulb</td>\n",
        "      <td> Wintergreen Lighting Christmas LED Light Bulb ...</td>\n",
        "      <td> WTGR1011\\nFeatures\\nNickel base, 60,000 averag...</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.471</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "   id                      query  \\\n",
        "0   1  bridal shower decorations   \n",
        "1   2       led christmas lights   \n",
        "2   4                  projector   \n",
        "3   5                  wine rack   \n",
        "4   7                 light bulb   \n",
        "\n",
        "                                       product_title  \\\n",
        "0        Accent Pillow with Heart Design - Red/Black   \n",
        "1  Set of 10 Battery Operated Multi LED Train Chr...   \n",
        "2         ViewSonic Pro8200 DLP Multimedia Projector   \n",
        "3  Concept Housewares WR-44526 Solid-Wood Ceiling...   \n",
        "4  Wintergreen Lighting Christmas LED Light Bulb ...   \n",
        "\n",
        "                                 product_description  median_relevance  \\\n",
        "0  Red satin accent pillow embroidered with a hea...                 1   \n",
        "1  Set of 10 Battery Operated Train Christmas Lig...                 4   \n",
        "2                                                                    4   \n",
        "3  Like a silent and sturdy tree, the Southern En...                 4   \n",
        "4  WTGR1011\\nFeatures\\nNickel base, 60,000 averag...                 2   \n",
        "\n",
        "   relevance_variance  \n",
        "0               0.000  \n",
        "1               0.000  \n",
        "2               0.471  \n",
        "3               0.000  \n",
        "4               0.471  "
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train[\"product_description\"][3]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "'Like a silent and sturdy tree, the Southern Enterprises Bird and Branch Coat Rack is an eye-catching addition to your home d\\xc3\\xa9cor. This tree themed coat rack features strong branches with pinecone accents and a small bird perched at the top to give it a whimsical and welcoming appearance while still making it sturdy enough to hold your coats, hats, umbrellas and more. Whether it serves as a coat rack, a hat rack or a combination of the two, it\\xe2\\x80\\x99ll be a great space saver that gets appreciated for its graceful appearance.\\nNumber of Hooks: 10\\nFrame Material: Metal\\nHardware Material: Metal\\nDimensions: 69.5 \" H x 13.25 \" W x 13.25 \" D\\nWeight: 12.0 Lb.\\nAssembly Details: assembly required'"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>id</th>\n",
        "      <th>query</th>\n",
        "      <th>product_title</th>\n",
        "      <th>product_description</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td>  3</td>\n",
        "      <td>               electric griddle</td>\n",
        "      <td>                   Star-Max 48 in Electric Griddle</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>  6</td>\n",
        "      <td>          phillips coffee maker</td>\n",
        "      <td> Philips SENSEO HD7810 WHITE Single Serve Pod C...</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>  9</td>\n",
        "      <td>            san francisco 49ers</td>\n",
        "      <td>                    2013 San Francisco 49ers Clock</td>\n",
        "      <td> A 2013 San Francisco 49ers clock is the ultima...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 11</td>\n",
        "      <td>                 aveeno shampoo</td>\n",
        "      <td>               AVEENO       10.5FLOZ NRSH SHINE SH</td>\n",
        "      <td> Water, Ammonium Lauryl Sulfate, Dimethicone, S...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 12</td>\n",
        "      <td> flea and tick control for dogs</td>\n",
        "      <td> Merial Frontline Plus Flea and Tick Control fo...</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "   id                           query  \\\n",
        "0   3                electric griddle   \n",
        "1   6           phillips coffee maker   \n",
        "2   9             san francisco 49ers   \n",
        "3  11                  aveeno shampoo   \n",
        "4  12  flea and tick control for dogs   \n",
        "\n",
        "                                       product_title  \\\n",
        "0                    Star-Max 48 in Electric Griddle   \n",
        "1  Philips SENSEO HD7810 WHITE Single Serve Pod C...   \n",
        "2                     2013 San Francisco 49ers Clock   \n",
        "3                AVEENO       10.5FLOZ NRSH SHINE SH   \n",
        "4  Merial Frontline Plus Flea and Tick Control fo...   \n",
        "\n",
        "                                 product_description  \n",
        "0                                                     \n",
        "1                                                     \n",
        "2  A 2013 San Francisco 49ers clock is the ultima...  \n",
        "3  Water, Ammonium Lauryl Sulfate, Dimethicone, S...  \n",
        "4                                                     "
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train.info()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<class 'pandas.core.frame.DataFrame'>\n",
        "Int64Index: 10158 entries, 0 to 10157\n",
        "Data columns (total 6 columns):\n",
        "id                     10158 non-null int64\n",
        "query                  10158 non-null object\n",
        "product_title          10158 non-null object\n",
        "product_description    10158 non-null object\n",
        "median_relevance       10158 non-null int64\n",
        "relevance_variance     10158 non-null float64\n",
        "dtypes: float64(1), int64(2), object(3)"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.info()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<class 'pandas.core.frame.DataFrame'>\n",
        "Int64Index: 22513 entries, 0 to 22512\n",
        "Data columns (total 4 columns):\n",
        "id                     22513 non-null int64\n",
        "query                  22513 non-null object\n",
        "product_title          22513 non-null object\n",
        "product_description    22513 non-null object\n",
        "dtypes: int64(1), object(3)"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>id</th>\n",
        "      <th>median_relevance</th>\n",
        "      <th>relevance_variance</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 10158.000000</td>\n",
        "      <td> 10158.000000</td>\n",
        "      <td> 10158.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td> 16353.103071</td>\n",
        "      <td>     3.309805</td>\n",
        "      <td>     0.377863</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>  9447.106683</td>\n",
        "      <td>     0.980666</td>\n",
        "      <td>     0.389707</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>     1.000000</td>\n",
        "      <td>     1.000000</td>\n",
        "      <td>     0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>  8078.750000</td>\n",
        "      <td>     3.000000</td>\n",
        "      <td>     0.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td> 16349.500000</td>\n",
        "      <td>     4.000000</td>\n",
        "      <td>     0.471000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td> 24570.750000</td>\n",
        "      <td>     4.000000</td>\n",
        "      <td>     0.471000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td> 32668.000000</td>\n",
        "      <td>     4.000000</td>\n",
        "      <td>     1.470000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 8,
       "text": [
        "                 id  median_relevance  relevance_variance\n",
        "count  10158.000000      10158.000000        10158.000000\n",
        "mean   16353.103071          3.309805            0.377863\n",
        "std     9447.106683          0.980666            0.389707\n",
        "min        1.000000          1.000000            0.000000\n",
        "25%     8078.750000          3.000000            0.000000\n",
        "50%    16349.500000          4.000000            0.471000\n",
        "75%    24570.750000          4.000000            0.471000\n",
        "max    32668.000000          4.000000            1.470000"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.describe()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>id</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>count</th>\n",
        "      <td> 22513.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>mean</th>\n",
        "      <td> 16328.282992</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>std</th>\n",
        "      <td>  9424.576451</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>min</th>\n",
        "      <td>     3.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>25%</th>\n",
        "      <td>  8201.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>50%</th>\n",
        "      <td> 16329.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>75%</th>\n",
        "      <td> 24464.000000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>max</th>\n",
        "      <td> 32671.000000</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "                 id\n",
        "count  22513.000000\n",
        "mean   16328.282992\n",
        "std     9424.576451\n",
        "min        3.000000\n",
        "25%     8201.000000\n",
        "50%    16329.000000\n",
        "75%    24464.000000\n",
        "max    32671.000000"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train[\"query\"].nunique()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 10,
       "text": [
        "261"
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test[\"query\"].nunique()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "261"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#All the queries that are in the training set, are available in the test set too!"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#set symmetric difference https://docs.python.org/2/library/sets.html\n",
      "#all elements in training query set are in the test query set\n",
      "train_unique_list=df_train[\"query\"].unique().tolist()\n",
      "test_unique_list=df_train[\"query\"].unique().tolist()\n",
      "set(train_unique_list)^set(test_unique_list)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 12,
       "text": [
        "set()"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#analyze product titles training set\n",
      "uniqueProdTitles=df_train[\"product_title\"].unique().tolist()\n",
      "idx=1\n",
      "for p in uniqueProdTitles[0:10]:\n",
      "    print str(idx) + \".\" + p\n",
      "    idx=idx+1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1.Accent Pillow with Heart Design - Red/Black\n",
        "2.Set of 10 Battery Operated Multi LED Train Christmas Lights - Clear Wire\n",
        "3.ViewSonic Pro8200 DLP Multimedia Projector\n",
        "4.Concept Housewares WR-44526 Solid-Wood Ceiling/Wall-Mount Wine Rack, Charcoal Grey, 6 Bottle\n",
        "5.Wintergreen Lighting Christmas LED Light Bulb (Pack of 25)\n",
        "6.Oakley Sunglasses - Radar Path Polished Black/Gray Sunglasses\n",
        "7.How To Make An American Quilt (DVD)\n",
        "8.ZAGG InvisibleShield Cell Phone Screen Protector for Samsung Galaxy S4 Mini\n",
        "9.Cook N Home Stainless Steel 4-Piece Pasta Cooker/ Steamer Multi-pots with Encapsulated Bottom, 8-Quart\n",
        "10.Presto FlipSide Electric Waffle Maker- 03510\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print \"Number of Unique product titles(train):\" + str(len(uniqueProdTitles))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Number of Unique product titles(train):9708\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#analyze product titles training set\n",
      "uniqueProdTitlesTest=df_test[\"product_title\"].unique().tolist()\n",
      "idx=1\n",
      "for p in uniqueProdTitlesTest[0:10]:\n",
      "    print str(idx) + \".\" + p\n",
      "    idx=idx+1"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1.Star-Max 48 in Electric Griddle\n",
        "2.Philips SENSEO HD7810 WHITE Single Serve Pod Coffee Maker Espresso Brew Machine\n",
        "3.2013 San Francisco 49ers Clock\n",
        "4.AVEENO       10.5FLOZ NRSH SHINE SH\n",
        "5.Merial Frontline Plus Flea and Tick Control for Dogs and Puppies 45 - 88 pound\n",
        "6.Classy Wood Table Clock\n",
        "7.Women's Infinity Scarf - Tribal Paisley\n",
        "8.Women's Skechers Flex Tech Tee Shirt White\n",
        "9.LEGO Star Wars Anakin's Jedi Interceptor 9494 - FACTORY SEALED - RETIRED\n",
        "10.Hamilton Beach Power Deluxe 4-Quart Stand Mixer\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print \"Number of Unique product titles(test):\" + str(len(uniqueProdTitlesTest))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Number of Unique product titles(test):21110\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#number of product titles only in the training set or only in the test set\n",
      "len(set(uniqueProdTitles)-set(uniqueProdTitlesTest))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "8680"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#intersection of product titles in the train and test sets\n",
      "len(set(uniqueProdTitles).intersection(uniqueProdTitlesTest))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "1028"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#The product titles don't really overlap (a few do though!)..."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Data preprocessing\n",
      "1. Parse query and titles using a regular expression very kindly put up in a starter script \n",
      "2. Convert to lower case\n",
      "3. Stem the words using the NLTK porter stemmer"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "stemmer=nltk.PorterStemmer() \n",
      "token_pattern = re.compile(r\"(?u)\\b\\w\\w+\\b\", re.UNICODE | re.LOCALE )\n",
      "query = []\n",
      "wordList = [] \n",
      "for i, row in df_train.iterrows():\n",
      "    query = [str(stemmer.stem(x.lower())) for x in token_pattern.findall(row[\"query\"])]\n",
      "    df_train.set_value(i,\"query\",' '.join(query))  \n",
      "    title = [str(stemmer.stem(x.lower())) for x in token_pattern.findall(row[\"product_title\"])]\n",
      "    df_train.set_value(i,\"product_title\",' '.join(title))  \n",
      "    description = [str(stemmer.stem(x.lower())) for x in token_pattern.findall(row[\"product_description\"])]\n",
      "    df_train.set_value(i,\"product_description\",' '.join(description))\n",
      " \n",
      "for i, row in df_test.iterrows():\n",
      "    query = [str(stemmer.stem(x.lower())) for x in token_pattern.findall(row[\"query\"])]\n",
      "    df_test.set_value(i,\"query\",' '.join(query))  \n",
      "    title = [str(stemmer.stem(x.lower())) for x in token_pattern.findall(row[\"product_title\"])]\n",
      "    df_test.set_value(i,\"product_title\",' '.join(title))  \n",
      "    description = [str(stemmer.stem(x.lower())) for x in token_pattern.findall(row[\"product_description\"])]\n",
      "    df_test.set_value(i,\"product_description\",' '.join(description))    \n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#The data after some cleaning up"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_train.head() "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>id</th>\n",
        "      <th>query</th>\n",
        "      <th>product_title</th>\n",
        "      <th>product_description</th>\n",
        "      <th>median_relevance</th>\n",
        "      <th>relevance_variance</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td> 1</td>\n",
        "      <td> bridal shower decor</td>\n",
        "      <td>         accent pillow with heart design red black</td>\n",
        "      <td> red satin accent pillow embroid with heart in ...</td>\n",
        "      <td> 1</td>\n",
        "      <td> 0.000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td> 2</td>\n",
        "      <td>  led christma light</td>\n",
        "      <td> set of 10 batteri oper multi led train christm...</td>\n",
        "      <td> set of 10 batteri oper train christma light it...</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0.000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td> 4</td>\n",
        "      <td>           projector</td>\n",
        "      <td>          viewson pro8200 dlp multimedia projector</td>\n",
        "      <td>                                                  </td>\n",
        "      <td> 4</td>\n",
        "      <td> 0.471</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 5</td>\n",
        "      <td>           wine rack</td>\n",
        "      <td> concept housewar wr 44526 solid wood ceil wall...</td>\n",
        "      <td> like silent and sturdi tree the southern enter...</td>\n",
        "      <td> 4</td>\n",
        "      <td> 0.000</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 7</td>\n",
        "      <td>          light bulb</td>\n",
        "      <td> wintergreen light christma led light bulb pack...</td>\n",
        "      <td> wtgr1011 featur nickel base 60 000 averag hour...</td>\n",
        "      <td> 2</td>\n",
        "      <td> 0.471</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 24,
       "text": [
        "   id                query                                      product_title  \\\n",
        "0   1  bridal shower decor          accent pillow with heart design red black   \n",
        "1   2   led christma light  set of 10 batteri oper multi led train christm...   \n",
        "2   4            projector           viewson pro8200 dlp multimedia projector   \n",
        "3   5            wine rack  concept housewar wr 44526 solid wood ceil wall...   \n",
        "4   7           light bulb  wintergreen light christma led light bulb pack...   \n",
        "\n",
        "                                 product_description  median_relevance  \\\n",
        "0  red satin accent pillow embroid with heart in ...                 1   \n",
        "1  set of 10 batteri oper train christma light it...                 4   \n",
        "2                                                                    4   \n",
        "3  like silent and sturdi tree the southern enter...                 4   \n",
        "4  wtgr1011 featur nickel base 60 000 averag hour...                 2   \n",
        "\n",
        "   relevance_variance  \n",
        "0               0.000  \n",
        "1               0.000  \n",
        "2               0.471  \n",
        "3               0.000  \n",
        "4               0.471  "
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "df_test.head()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>id</th>\n",
        "      <th>query</th>\n",
        "      <th>product_title</th>\n",
        "      <th>product_description</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>0</th>\n",
        "      <td>  3</td>\n",
        "      <td>                 electr griddl</td>\n",
        "      <td>                      star max 48 in electr griddl</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1</th>\n",
        "      <td>  6</td>\n",
        "      <td>           phillip coffe maker</td>\n",
        "      <td> philip senseo hd7810 white singl serv pod coff...</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>2</th>\n",
        "      <td>  9</td>\n",
        "      <td>            san francisco 49er</td>\n",
        "      <td>                     2013 san francisco 49er clock</td>\n",
        "      <td> 2013 san francisco 49er clock is the ultim way...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>3</th>\n",
        "      <td> 11</td>\n",
        "      <td>                aveeno shampoo</td>\n",
        "      <td>                     aveeno 10 5floz nrsh shine sh</td>\n",
        "      <td> water ammonium lauryl sulfat dimethicon sodium...</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>4</th>\n",
        "      <td> 12</td>\n",
        "      <td> flea and tick control for dog</td>\n",
        "      <td> merial frontlin plu flea and tick control for ...</td>\n",
        "      <td>                                                  </td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 42,
       "text": [
        "   id                          query  \\\n",
        "0   3                  electr griddl   \n",
        "1   6            phillip coffe maker   \n",
        "2   9             san francisco 49er   \n",
        "3  11                 aveeno shampoo   \n",
        "4  12  flea and tick control for dog   \n",
        "\n",
        "                                       product_title  \\\n",
        "0                       star max 48 in electr griddl   \n",
        "1  philip senseo hd7810 white singl serv pod coff...   \n",
        "2                      2013 san francisco 49er clock   \n",
        "3                      aveeno 10 5floz nrsh shine sh   \n",
        "4  merial frontlin plu flea and tick control for ...   \n",
        "\n",
        "                                 product_description  \n",
        "0                                                     \n",
        "1                                                     \n",
        "2  2013 san francisco 49er clock is the ultim way...  \n",
        "3  water ammonium lauryl sulfat dimethicon sodium...  \n",
        "4                                                     "
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#compile the query and product titles from the training and test sets\n",
      "trainQueryList = df_train[\"query\"].tolist()\n",
      "trainTitleList = df_train[\"product_title\"].tolist()\n",
      "queryTitleTrain = []\n",
      "\n",
      "for query,title in zip(trainQueryList,trainTitleList):\n",
      "    sdatalist = []\n",
      "    sdatalist.append(query)\n",
      "    sdatalist.append(title)\n",
      "    queryTitleTrain.append(\" \".join(sdatalist))\n",
      "    \n",
      "\n",
      "\n",
      "\n",
      "testQueryList = df_test[\"query\"].tolist()\n",
      "testTitleList = df_test[\"product_title\"].tolist()\n",
      "queryTitleTest = []\n",
      "for query,title in zip(testQueryList,testTitleList):\n",
      "    sdatalist = []\n",
      "    sdatalist.append(query)\n",
      "    sdatalist.append(title)\n",
      "    queryTitleTest.append(\" \".join(sdatalist))\n",
      "\n",
      "#combine the train and test sets to vectorize\n",
      "combinedList = queryTitleTrain + queryTitleTest    \n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Use the TF-IDF vectorizer to prepare training and test vectors -- SKLearn to the rescue!\n",
      "\n",
      "--targets are the median_relevance values"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tfv = text.TfidfVectorizer(min_df=3, max_df=500, max_features=None, strip_accents='unicode', analyzer='word', token_pattern=r'\\w{1,}', ngram_range=(1, 2),\\\n",
      " use_idf=True, smooth_idf=True, sublinear_tf=True, stop_words = 'english')\n",
      "\n",
      "tfv.fit(combinedList)\n",
      "\n",
      "XTrain = tfv.transform(queryTitleTrain)\n",
      "XTest = tfv.transform(queryTitleTest)\n",
      "y=np.array(df_train[\"median_relevance\"].tolist())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Use a cascade of operations and Voila!\n",
      "1. TF-IDF vectors are sparse and high-dimensional\n",
      "2. Directly fitting it to any sort of classifier or regressor invloves fitting a large number of parameters\n",
      "3. Since 4 is the most commonly occurring median relevance value, if you overfit...it looks like you might as well have typed out the number 4 ~20k times! And your score on using the weighted Kappa metric is 0.0! (Man all that work!) \n",
      "4. One common way to reduce overfitting is to use dimensionality reduction, Principal components analysis which uses a Singular Value decomposition by-product (the right and left matrices of which are orthogonal eigen vectors) is a good start! "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf = Pipeline([('svd', TruncatedSVD(n_components=400, algorithm='randomized', n_iter=5, random_state=None, tol=0.0)), ('scl', StandardScaler(copy=Tr\\\n",
      "ue, with_mean=True, with_std=True)),('svm', SVC(C=10.0, kernel='rbf', degree=3, gamma=0.0, coef0=0.0, shrinking=True, probability=False, tol=0.001, \\\n",
      "cache_size=200, class_weight=None, verbose=False, max_iter=-1, random_state=None))])\n",
      "\n",
      "clf.fit(XTrain, y) \n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "Pipeline(steps=[('svd', TruncatedSVD(algorithm='randomized', n_components=400, n_iter=5,\n",
        "       random_state=None, tol=0.0)), ('scl', StandardScaler(copy=True, with_mean=True, with_std=True)), ('svm', SVC(C=10.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.0,\n",
        "  kernel='rbf', max_iter=-1, probability=False, random_state=None,\n",
        "  shrinking=True, tol=0.001, verbose=False))])"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "t_labels=clf.predict(XTest)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#output results for submission                                                                                                                        \n",
      "with open(\"submission_full_tfidf_ak.csv\",\"w\") as f:\n",
      "    f.write(\"id,prediction\\n\")\n",
      "    for i in range(len(t_labels)):\n",
      "        f.write(str(df_test.id[i])+\",\"+str(t_labels[i])+\"\\n\")\n",
      "f.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Kaggle score (with SVD): 0.57749\n",
      "#Kaggle score (without SVD) : 0.0"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Fancier preprocessing, with product descriptions included - Taken from the link to the starter script linked above"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "sw=[]\n",
      "s_data = []\n",
      "s_labels = []\n",
      "t_data = []\n",
      "t_labels = []\n",
      "combined_data = []\n",
      "#stopwords tweak - more overhead                                                                                                                      \n",
      "stop_words = ['http','www','img','border','0','1','2','3','4','5','6','7','8','9']\n",
      "stop_words = text.ENGLISH_STOP_WORDS.union(stop_words)\n",
      "for stw in stop_words:\n",
      "    sw.append(\"q\"+stw)\n",
      "    sw.append(\"z\"+stw)\n",
      "stop_words = text.ENGLISH_STOP_WORDS.union(sw)\n",
      "\n",
      "stemmer = nltk.PorterStemmer()\n",
      "\n",
      "for i in range(len(df_train.id)):\n",
      "    s=(\" \").join([\"q\"+ z for z in BeautifulSoup(df_train[\"query\"][i]).get_text(\" \").split(\" \")]) + \" \" + (\" \").join([\"z\"+ z for z in BeautifulSoup(df_train.product_title[i]).get_text(\" \").split(\" \")]) + \" \" + BeautifulSoup(df_train.product_description[i]).get_text(\" \")\n",
      "    s=re.sub(\"[^a-zA-Z0-9]\",\" \", s)\n",
      "    s= (\" \").join([stemmer.stem(z) for z in s.split(\" \")])\n",
      "    s_data.append(s)\n",
      "    s_labels.append(str(df_train[\"median_relevance\"][i]))\n",
      "\n",
      "for i in range(len(df_test.id)):\n",
      "    s=(\" \").join([\"q\"+ z for z in BeautifulSoup(df_test[\"query\"][i]).get_text().split(\" \")]) + \" \" + (\" \").join([\"z\"+ z for z in BeautifulSoup(df_test.prod\\\n",
      "uct_title[i]).get_text().split(\" \")]) + \" \" + BeautifulSoup(df_test.product_description[i]).get_text()\n",
      "    s=re.sub(\"[^a-zA-Z0-9]\",\" \", s)\n",
      "    s= (\" \").join([stemmer.stem(z) for z in s.split(\" \")])\n",
      "    t_data.append(s)\n",
      "    \n",
      "for i in range(len(df_train.id)):\n",
      "    combined_data.append(s_data[i])\n",
      "\n",
      "for i in range(len(df_test.id)):\n",
      "    combined_data.append(t_data[i])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/home/aanchan/anaconda/lib/python2.7/site-packages/bs4/__init__.py:189: UserWarning: \"http://i104.photobucket.com/albums/m175/champions_on_display/wincraft2013/januaryb/65497012.jpg\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
        "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n",
        "/home/aanchan/anaconda/lib/python2.7/site-packages/bs4/__init__.py:189: UserWarning: \"http://i104.photobucket.com/albums/m175/champions_on_display/wincraft2013/januaryb/65516012.jpg\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
        "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/home/aanchan/anaconda/lib/python2.7/site-packages/bs4/__init__.py:189: UserWarning: \"http://i104.photobucket.com/albums/m175/champions_on_display/wincraft2013/januaryb/6552101\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
        "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n",
        "/home/aanchan/anaconda/lib/python2.7/site-packages/bs4/__init__.py:189: UserWarning: \"http://i104.photobucket.com/albums/m175/champions_on_display/wincraft2013/januaryb/65527\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
        "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "/home/aanchan/anaconda/lib/python2.7/site-packages/bs4/__init__.py:189: UserWarning: \"http://i104.photobucket.com/albums/m175/champions_on_display/wincraft2013/januarya/14146012.jpg\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.\n",
        "  '\"%s\" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)\n"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "tfv = text.TfidfVectorizer(min_df=3, max_df=500, max_features=None, strip_accents='unicode', analyzer='word', token_pattern=r'\\w{1,}', ngram_range=(1, 2),\\\n",
      " use_idf=True, smooth_idf=True, sublinear_tf=True, stop_words = 'english')\n",
      "tfv.fit(combined_data)\n",
      "s_data = tfv.transform(s_data)\n",
      "t_data = tfv.transform(t_data)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf = Pipeline([('svd', TruncatedSVD(n_components=400, algorithm='randomized', n_iter=5, random_state=None, tol=0.0)), ('scl', StandardScaler(copy=Tr\\\n",
      "ue, with_mean=True, with_std=True)),('svm', SVC(C=10.0, kernel='rbf', degree=3, gamma=0.0, coef0=0.0, shrinking=True, probability=False, tol=0.001, \\\n",
      "cache_size=200, class_weight=None, verbose=False, max_iter=-1, random_state=None))])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "clf.fit(s_data, s_labels)                                                                                                                             \n",
      "t_labels = clf.predict(t_data)\n",
      "\n",
      "#output results for submission                                                                                                                        \n",
      "with open(\"submission_full_tfidf.csv\",\"w\") as f:\n",
      "    f.write(\"id,prediction\\n\")\n",
      "    for i in range(len(t_labels)):\n",
      "        f.write(str(df_test.id[i])+\",\"+str(t_labels[i])+\"\\n\")\n",
      "f.close()\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Kaggle score:0.59246"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Lets talk about overfitting\n"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<img src=\"overfit.jpg\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#List of things winners did not do\n",
      "1. Use additional data\n",
      "2. Use models that allow higher level representations that capture relationships between words"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Other links on recurrent neural networks and language models\n",
      "\n",
      "[Stanford Language Modelling - NGrams](https://web.stanford.edu/class/cs124/lec/languagemodeling.pdf)\n",
      "\n",
      "[Tomas Mikolov's presentation -RNNLM](http://www.fit.vutbr.cz/~imikolov/rnnlm/google.pdf)\n",
      "\n",
      "[Yandex presentation - Faster RNNLM](https://yadi.sk/d/8Hko3d7NkB5q5)\n",
      "\n",
      "[Tensorflow RNN Tutorial](https://www.tensorflow.org/versions/r0.9/tutorials/recurrent/index.html)\n",
      "\n",
      "[Tensorflow Stanford tutorial slides](https://cs224d.stanford.edu/lectures/CS224d-Lecture7.pdf)\n",
      "\n",
      "[Tensorflow MNIST Tutorial](https://github.com/aanchan/TensorFlow-Tutorials/blob/master/3_mnist_from_scratch.ipynb)\n",
      "\n",
      "[Tensorflow word2vec tutorial](https://www.tensorflow.org/versions/r0.9/tutorials/word2vec/index.html)\n",
      "\n",
      "[Word2Vec intuition](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/)\n",
      "\n",
      "[Oxford RNN lecture - Dr Nando de Frietas](https://www.youtube.com/watch?v=56TYLaQN4N8)\n",
      "\n",
      "[Kappa statistic in plain English](http://stats.stackexchange.com/questions/82162/kappa-statistic-in-plain-english)"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Tensorflow specific notes\n",
      "\n",
      "[Tensorflow installation instructions](https://www.tensorflow.org/versions/r0.9/get_started/os_setup.html#download-and-setup) -- You can install from source, using Anaconda and VirtualEnv or Docker. I found the Docker install to be the most painless.\n",
      "\n",
      "[Penn Tree Bank Data](http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz)\n",
      "\n",
      "The two scripts ptb_word_lm.py and reader.py the Tensorflow specific tutorial website refers to are available on the path tensorflow/tensorflow/models/rnn/ptb available from [Tensorflow github repo](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/models/rnn/ptb). The Docker image does does not seem to have ptb_word_lm.py. reader.py though is hidden away in /usr/local/lib/python2.7/dist-packages/tensorflow/models/rnn/ptb/reader.py in the Docker image.\n",
      "\n",
      "[Understand Tensorflow fetches and feeds](https://www.tensorflow.org/versions/r0.9/get_started/basic_usage.html)\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}