MovieRecommenderSystem.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2f871f5b",
   "metadata": {},
   "source": [
    "# Movie Recommender System on TMDB Dataset\n",
    "\n",
    "Some common types of recommendation systems:\n",
    "\n",
    "1. **Content-based filtering**: This approach analyzes item characteristics and user profiles to provide recommendations. It suggests items similar to the ones users have shown interest in, based on shared attributes such as genre or keywords.\n",
    "\n",
    "2. **Collaborative filtering**: By identifying patterns among users or items, collaborative filtering generates recommendations. It looks for similarities between users’ preferences or item ratings to suggest items that others with similar tastes have enjoyed.\n",
    "\n",
    "3. **Hybrid recommender systems**: These systems combine multiple techniques to enhance recommendation accuracy and diversity. They leverage the strengths of different approaches, such as content-based and collaborative filtering, to provide more personalized suggestions.\n",
    "\n",
    "4. **Knowledge-based systems**: Using explicit knowledge and rules, knowledge-based systems make recommendations based on specific criteria or constraints. They incorporate domain knowledge to suggest items that align with predefined rules or user preferences.\n",
    "\n",
    "5. **Popularity-based systems**: These systems rely on the overall popularity of products or items within a particular demographic. Recommendations are based on what is trending or frequently purchased by others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51a169f2",
   "metadata": {},
   "source": [
    "We will focus on building a **content-based recommender engine** for movies, where recommendations are based on the type of movies users search for.\n",
    "\n",
    "Dataset: TMDB 5000 dataset\n",
    "\n",
    "This dataset contains valuable information about movies, including their titles, genres, keywords, and user ratings. there are two csv files: one that contains credits and another brimming with movie details. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eff08488",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "67a7db87",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "378cfdf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_credits = pd.read_csv('tmdb_5000_credits.csv')\n",
    "df_movies = pd.read_csv('tmdb_5000_movies.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "010cf7d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4803, 4), (4803, 20))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_credits.shape, df_movies.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7e8972d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>movie_id</th>\n",
       "      <th>title</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   movie_id                                     title  \\\n",
       "0     19995                                    Avatar   \n",
       "1       285  Pirates of the Caribbean: At World's End   \n",
       "2    206647                                   Spectre   \n",
       "3     49026                     The Dark Knight Rises   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "\n",
       "                                                crew  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_credits.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9f9607b1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>popularity</th>\n",
       "      <th>production_companies</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>19995</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>150.437577</td>\n",
       "      <td>[{\"name\": \"Ingenious Film Partners\", \"id\": 289...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2009-12-10</td>\n",
       "      <td>2787965087</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>285</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>139.082615</td>\n",
       "      <td>[{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2007-05-19</td>\n",
       "      <td>961000000</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>245000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.sonypictures.com/movies/spectre/</td>\n",
       "      <td>206647</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>en</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>107.376788</td>\n",
       "      <td>[{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...</td>\n",
       "      <td>[{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...</td>\n",
       "      <td>2015-10-26</td>\n",
       "      <td>880674609</td>\n",
       "      <td>148.0</td>\n",
       "      <td>[{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...</td>\n",
       "      <td>Released</td>\n",
       "      <td>A Plan No One Escapes</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>4466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>250000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>http://www.thedarkknightrises.com/</td>\n",
       "      <td>49026</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>en</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>112.312950</td>\n",
       "      <td>[{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2012-07-16</td>\n",
       "      <td>1084939099</td>\n",
       "      <td>165.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>The Legend Ends</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>9106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      budget                                             genres  \\\n",
       "0  237000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  300000000  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "2  245000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "3  250000000  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...   \n",
       "\n",
       "                                       homepage      id  \\\n",
       "0                   http://www.avatarmovie.com/   19995   \n",
       "1  http://disney.go.com/disneypictures/pirates/     285   \n",
       "2   http://www.sonypictures.com/movies/spectre/  206647   \n",
       "3            http://www.thedarkknightrises.com/   49026   \n",
       "\n",
       "                                            keywords original_language  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...                en   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...                en   \n",
       "2  [{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...                en   \n",
       "3  [{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...                en   \n",
       "\n",
       "                             original_title  \\\n",
       "0                                    Avatar   \n",
       "1  Pirates of the Caribbean: At World's End   \n",
       "2                                   Spectre   \n",
       "3                     The Dark Knight Rises   \n",
       "\n",
       "                                            overview  popularity  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...  150.437577   \n",
       "1  Captain Barbossa, long believed to be dead, ha...  139.082615   \n",
       "2  A cryptic message from Bond’s past sends him o...  107.376788   \n",
       "3  Following the death of District Attorney Harve...  112.312950   \n",
       "\n",
       "                                production_companies  \\\n",
       "0  [{\"name\": \"Ingenious Film Partners\", \"id\": 289...   \n",
       "1  [{\"name\": \"Walt Disney Pictures\", \"id\": 2}, {\"...   \n",
       "2  [{\"name\": \"Columbia Pictures\", \"id\": 5}, {\"nam...   \n",
       "3  [{\"name\": \"Legendary Pictures\", \"id\": 923}, {\"...   \n",
       "\n",
       "                                production_countries release_date     revenue  \\\n",
       "0  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2009-12-10  2787965087   \n",
       "1  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2007-05-19   961000000   \n",
       "2  [{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...   2015-10-26   880674609   \n",
       "3  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...   2012-07-16  1084939099   \n",
       "\n",
       "   runtime                                   spoken_languages    status  \\\n",
       "0    162.0  [{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...  Released   \n",
       "1    169.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "2    148.0  [{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...  Released   \n",
       "3    165.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]  Released   \n",
       "\n",
       "                                          tagline  \\\n",
       "0                     Enter the World of Pandora.   \n",
       "1  At the end of the world, the adventure begins.   \n",
       "2                           A Plan No One Escapes   \n",
       "3                                 The Legend Ends   \n",
       "\n",
       "                                      title  vote_average  vote_count  \n",
       "0                                    Avatar           7.2       11800  \n",
       "1  Pirates of the Caribbean: At World's End           6.9        4500  \n",
       "2                                   Spectre           6.3        4466  \n",
       "3                     The Dark Knight Rises           7.6        9106  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_movies.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cbbfcda9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check whether the id column from df_movies is same as movie_id from df_credits\n",
    "(df_credits.movie_id != df_movies.id).any().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "aedc4571",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rename the column movie_id name in df_cred to id followed by merge\n",
    "df_credits.rename(columns = {'movie_id': 'id'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "baf2d4b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merge the two dataframes\n",
    "df = df_credits.merge(df_movies, on = 'id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "75225515",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title_x</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>keywords</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>...</th>\n",
       "      <th>production_countries</th>\n",
       "      <th>release_date</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title_y</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "      <td>237000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.avatarmovie.com/</td>\n",
       "      <td>[{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...</td>\n",
       "      <td>en</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2009-12-10</td>\n",
       "      <td>2787965087</td>\n",
       "      <td>162.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Enter the World of Pandora.</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>11800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "      <td>300000000</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>http://disney.go.com/disneypictures/pirates/</td>\n",
       "      <td>[{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...</td>\n",
       "      <td>en</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2007-05-19</td>\n",
       "      <td>961000000</td>\n",
       "      <td>169.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>At the end of the world, the adventure begins.</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>4500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "      <td>245000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>http://www.sonypictures.com/movies/spectre/</td>\n",
       "      <td>[{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...</td>\n",
       "      <td>en</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...</td>\n",
       "      <td>2015-10-26</td>\n",
       "      <td>880674609</td>\n",
       "      <td>148.0</td>\n",
       "      <td>[{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...</td>\n",
       "      <td>Released</td>\n",
       "      <td>A Plan No One Escapes</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>4466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "      <td>250000000</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>http://www.thedarkknightrises.com/</td>\n",
       "      <td>[{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...</td>\n",
       "      <td>en</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>...</td>\n",
       "      <td>[{\"iso_3166_1\": \"US\", \"name\": \"United States o...</td>\n",
       "      <td>2012-07-16</td>\n",
       "      <td>1084939099</td>\n",
       "      <td>165.0</td>\n",
       "      <td>[{\"iso_639_1\": \"en\", \"name\": \"English\"}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>The Legend Ends</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>9106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                   title_x  \\\n",
       "0   19995                                    Avatar   \n",
       "1     285  Pirates of the Caribbean: At World's End   \n",
       "2  206647                                   Spectre   \n",
       "3   49026                     The Dark Knight Rises   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "\n",
       "                                                crew     budget  \\\n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...  237000000   \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...  300000000   \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...  245000000   \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...  250000000   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "2  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "3  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...   \n",
       "\n",
       "                                       homepage  \\\n",
       "0                   http://www.avatarmovie.com/   \n",
       "1  http://disney.go.com/disneypictures/pirates/   \n",
       "2   http://www.sonypictures.com/movies/spectre/   \n",
       "3            http://www.thedarkknightrises.com/   \n",
       "\n",
       "                                            keywords original_language  \\\n",
       "0  [{\"id\": 1463, \"name\": \"culture clash\"}, {\"id\":...                en   \n",
       "1  [{\"id\": 270, \"name\": \"ocean\"}, {\"id\": 726, \"na...                en   \n",
       "2  [{\"id\": 470, \"name\": \"spy\"}, {\"id\": 818, \"name...                en   \n",
       "3  [{\"id\": 849, \"name\": \"dc comics\"}, {\"id\": 853,...                en   \n",
       "\n",
       "                             original_title  ...  \\\n",
       "0                                    Avatar  ...   \n",
       "1  Pirates of the Caribbean: At World's End  ...   \n",
       "2                                   Spectre  ...   \n",
       "3                     The Dark Knight Rises  ...   \n",
       "\n",
       "                                production_countries  release_date  \\\n",
       "0  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...    2009-12-10   \n",
       "1  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...    2007-05-19   \n",
       "2  [{\"iso_3166_1\": \"GB\", \"name\": \"United Kingdom\"...    2015-10-26   \n",
       "3  [{\"iso_3166_1\": \"US\", \"name\": \"United States o...    2012-07-16   \n",
       "\n",
       "      revenue runtime                                   spoken_languages  \\\n",
       "0  2787965087   162.0  [{\"iso_639_1\": \"en\", \"name\": \"English\"}, {\"iso...   \n",
       "1   961000000   169.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]   \n",
       "2   880674609   148.0  [{\"iso_639_1\": \"fr\", \"name\": \"Fran\\u00e7ais\"},...   \n",
       "3  1084939099   165.0           [{\"iso_639_1\": \"en\", \"name\": \"English\"}]   \n",
       "\n",
       "     status                                         tagline  \\\n",
       "0  Released                     Enter the World of Pandora.   \n",
       "1  Released  At the end of the world, the adventure begins.   \n",
       "2  Released                           A Plan No One Escapes   \n",
       "3  Released                                 The Legend Ends   \n",
       "\n",
       "                                    title_y vote_average vote_count  \n",
       "0                                    Avatar          7.2      11800  \n",
       "1  Pirates of the Caribbean: At World's End          6.9       4500  \n",
       "2                                   Spectre          6.3       4466  \n",
       "3                     The Dark Knight Rises          7.6       9106  \n",
       "\n",
       "[4 rows x 23 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ebde4d61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4803 entries, 0 to 4802\n",
      "Data columns (total 23 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   id                    4803 non-null   int64  \n",
      " 1   title_x               4803 non-null   object \n",
      " 2   cast                  4803 non-null   object \n",
      " 3   crew                  4803 non-null   object \n",
      " 4   budget                4803 non-null   int64  \n",
      " 5   genres                4803 non-null   object \n",
      " 6   homepage              1712 non-null   object \n",
      " 7   keywords              4803 non-null   object \n",
      " 8   original_language     4803 non-null   object \n",
      " 9   original_title        4803 non-null   object \n",
      " 10  overview              4800 non-null   object \n",
      " 11  popularity            4803 non-null   float64\n",
      " 12  production_companies  4803 non-null   object \n",
      " 13  production_countries  4803 non-null   object \n",
      " 14  release_date          4802 non-null   object \n",
      " 15  revenue               4803 non-null   int64  \n",
      " 16  runtime               4801 non-null   float64\n",
      " 17  spoken_languages      4803 non-null   object \n",
      " 18  status                4803 non-null   object \n",
      " 19  tagline               3959 non-null   object \n",
      " 20  title_y               4803 non-null   object \n",
      " 21  vote_average          4803 non-null   float64\n",
      " 22  vote_count            4803 non-null   int64  \n",
      "dtypes: float64(3), int64(4), object(16)\n",
      "memory usage: 900.6+ KB\n"
     ]
    }
   ],
   "source": [
    "# Lets take a look at the data type of the columns\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ac237252",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                         0\n",
       "title_x                    0\n",
       "cast                       0\n",
       "crew                       0\n",
       "budget                     0\n",
       "genres                     0\n",
       "homepage                3091\n",
       "keywords                   0\n",
       "original_language          0\n",
       "original_title             0\n",
       "overview                   3\n",
       "popularity                 0\n",
       "production_companies       0\n",
       "production_countries       0\n",
       "release_date               1\n",
       "revenue                    0\n",
       "runtime                    2\n",
       "spoken_languages           0\n",
       "status                     0\n",
       "tagline                  844\n",
       "title_y                    0\n",
       "vote_average               0\n",
       "vote_count                 0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7ea173",
   "metadata": {},
   "source": [
    "By combining elements such as the movie title, genre, overview, and cast/crew (if we desire to search based on individuals), we can generate a cohesive representation that will enable us to discover similar movies effectively.\n",
    "\n",
    "Since we are considering the Overview column and there are 3 null values, lets drop those null values first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "158df6ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop null overviews\n",
    "df.dropna(subset = ['overview'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d1fca663",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter out target columns\n",
    "df = df[['id', 'title_x', 'genres', 'overview', 'cast', 'crew','vote_average']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "54868a60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4800 entries, 0 to 4802\n",
      "Data columns (total 7 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   id            4800 non-null   int64  \n",
      " 1   title_x       4800 non-null   object \n",
      " 2   genres        4800 non-null   object \n",
      " 3   overview      4800 non-null   object \n",
      " 4   cast          4800 non-null   object \n",
      " 5   crew          4800 non-null   object \n",
      " 6   vote_average  4800 non-null   float64\n",
      "dtypes: float64(1), int64(1), object(5)\n",
      "memory usage: 429.0+ KB\n"
     ]
    }
   ],
   "source": [
    "# check new df info\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e2ff914a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title_x</th>\n",
       "      <th>genres</th>\n",
       "      <th>overview</th>\n",
       "      <th>cast</th>\n",
       "      <th>crew</th>\n",
       "      <th>vote_average</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "      <td>[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...</td>\n",
       "      <td>[{\"credit_id\": \"52fe48009251416c750aca23\", \"de...</td>\n",
       "      <td>7.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>[{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "      <td>[{\"cast_id\": 4, \"character\": \"Captain Jack Spa...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...</td>\n",
       "      <td>6.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "      <td>[{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...</td>\n",
       "      <td>[{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...</td>\n",
       "      <td>6.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "      <td>[{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...</td>\n",
       "      <td>[{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...</td>\n",
       "      <td>7.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "      <td>[{\"cast_id\": 5, \"character\": \"John Carter\", \"c...</td>\n",
       "      <td>[{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...</td>\n",
       "      <td>6.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                   title_x  \\\n",
       "0   19995                                    Avatar   \n",
       "1     285  Pirates of the Caribbean: At World's End   \n",
       "2  206647                                   Spectre   \n",
       "3   49026                     The Dark Knight Rises   \n",
       "4   49529                               John Carter   \n",
       "\n",
       "                                              genres  \\\n",
       "0  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "1  [{\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"...   \n",
       "2  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "3  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 80, \"nam...   \n",
       "4  [{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"nam...   \n",
       "\n",
       "                                            overview  \\\n",
       "0  In the 22nd century, a paraplegic Marine is di...   \n",
       "1  Captain Barbossa, long believed to be dead, ha...   \n",
       "2  A cryptic message from Bond’s past sends him o...   \n",
       "3  Following the death of District Attorney Harve...   \n",
       "4  John Carter is a war-weary, former military ca...   \n",
       "\n",
       "                                                cast  \\\n",
       "0  [{\"cast_id\": 242, \"character\": \"Jake Sully\", \"...   \n",
       "1  [{\"cast_id\": 4, \"character\": \"Captain Jack Spa...   \n",
       "2  [{\"cast_id\": 1, \"character\": \"James Bond\", \"cr...   \n",
       "3  [{\"cast_id\": 2, \"character\": \"Bruce Wayne / Ba...   \n",
       "4  [{\"cast_id\": 5, \"character\": \"John Carter\", \"c...   \n",
       "\n",
       "                                                crew  vote_average  \n",
       "0  [{\"credit_id\": \"52fe48009251416c750aca23\", \"de...           7.2  \n",
       "1  [{\"credit_id\": \"52fe4232c3a36847f800b579\", \"de...           6.9  \n",
       "2  [{\"credit_id\": \"54805967c3a36829b5002c41\", \"de...           6.3  \n",
       "3  [{\"credit_id\": \"52fe4781c3a36847f81398c3\", \"de...           7.6  \n",
       "4  [{\"credit_id\": \"52fe479ac3a36847f813eaa3\", \"de...           6.1  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc1ca415",
   "metadata": {},
   "source": [
    "**overview** and **title** columns contain simple string values, which makes them straightforward to handle.\n",
    "On the other hand, the **genres, cast, and crew** columns follow a similar structure, as they consist of lists of dictionaries.\n",
    "\n",
    "To extract the desired values from each dictionary, we need to define a specific approach. For genres, we can include almost all genres associated with each movie as relevant tags. \n",
    "\n",
    "However, when it comes to the cast, including the entire list would lead to increased variance, so we will only add the top 4(you can vary) cast members. Similarly, for the crew, we will focus on directors and producers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a78b2839",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"id\": 28, \"name\": \"Action\"}, {\"id\": 12, \"name\": \"Adventure\"}, {\"id\": 14, \"name\": \"Fantasy\"}, {\"id\": 878, \"name\": \"Science Fiction\"}]'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.genres[0] # we will include all the \"name\" tags from Genres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0f7dc87c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Action Adventure Fantasy Science Fiction'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([i['name'] for i in eval(df.genres[0])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e4d95936",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"cast_id\": 242, \"character\": \"Jake Sully\", \"credit_id\": \"5602a8a7c3a3685532001c9a\", \"gender\": 2, \"id\": 65731, \"name\": \"Sam Worthington\", \"order\": 0}, {\"cast_id\": 3, \"character\": \"Neytiri\", \"credit_id\": \"52fe48009251416c750ac9cb\", \"gender\": 1, \"id\": 8691, \"name\": \"Zoe Saldana\", \"order\": 1}, {\"cast_id\": 25, \"character\": \"Dr. Grace Augustine\", \"credit_id\": \"52fe48009251416c750aca39\", \"gender\": 1, \"id\": 10205, \"name\": \"Sigourney Weaver\", \"order\": 2}, {\"cast_id\": 4, \"character\": \"Col. Quaritch\", \"credit_id\": \"52fe48009251416c750ac9cf\", \"gender\": 2, \"id\": 32747, \"name\": \"Stephen Lang\", \"order\": 3}, {\"cast_id\": 5, \"character\": \"Trudy Chacon\", \"credit_id\": \"52fe48009251416c750ac9d3\", \"gender\": 1, \"id\": 17647, \"name\": \"Michelle Rodriguez\", \"order\": 4}, {\"cast_id\": 8, \"character\": \"Selfridge\", \"credit_id\": \"52fe48009251416c750ac9e1\", \"gender\": 2, \"id\": 1771, \"name\": \"Giovanni Ribisi\", \"order\": 5}, {\"cast_id\": 7, \"character\": \"Norm Spellman\", \"credit_id\": \"52fe48009251416c750ac9dd\", \"gender\": 2, \"id\": 59231, \"name\": \"Joel David Moore\", \"order\": 6}, {\"cast_id\": 9, \"character\": \"Moat\", \"credit_id\": \"52fe48009251416c750ac9e5\", \"gender\": 1, \"id\": 30485, \"name\": \"CCH Pounder\", \"order\": 7}, {\"cast_id\": 11, \"character\": \"Eytukan\", \"credit_id\": \"52fe48009251416c750ac9ed\", \"gender\": 2, \"id\": 15853, \"name\": \"Wes Studi\", \"order\": 8}, {\"cast_id\": 10, \"character\": \"Tsu\\'Tey\", \"credit_id\": \"52fe48009251416c750ac9e9\", \"gender\": 2, \"id\": 10964, \"name\": \"Laz Alonso\", \"order\": 9}, {\"cast_id\": 12, \"character\": \"Dr. Max Patel\", \"credit_id\": \"52fe48009251416c750ac9f1\", \"gender\": 2, \"id\": 95697, \"name\": \"Dileep Rao\", \"order\": 10}, {\"cast_id\": 13, \"character\": \"Lyle Wainfleet\", \"credit_id\": \"52fe48009251416c750ac9f5\", \"gender\": 2, \"id\": 98215, \"name\": \"Matt Gerald\", \"order\": 11}, {\"cast_id\": 32, \"character\": \"Private Fike\", \"credit_id\": \"52fe48009251416c750aca5b\", \"gender\": 2, \"id\": 154153, \"name\": \"Sean Anthony Moran\", \"order\": 12}, {\"cast_id\": 33, \"character\": \"Cryo Vault Med Tech\", \"credit_id\": \"52fe48009251416c750aca5f\", \"gender\": 2, \"id\": 397312, \"name\": \"Jason Whyte\", \"order\": 13}, {\"cast_id\": 34, \"character\": \"Venture Star Crew Chief\", \"credit_id\": \"52fe48009251416c750aca63\", \"gender\": 2, \"id\": 42317, \"name\": \"Scott Lawrence\", \"order\": 14}, {\"cast_id\": 35, \"character\": \"Lock Up Trooper\", \"credit_id\": \"52fe48009251416c750aca67\", \"gender\": 2, \"id\": 986734, \"name\": \"Kelly Kilgour\", \"order\": 15}, {\"cast_id\": 36, \"character\": \"Shuttle Pilot\", \"credit_id\": \"52fe48009251416c750aca6b\", \"gender\": 0, \"id\": 1207227, \"name\": \"James Patrick Pitt\", \"order\": 16}, {\"cast_id\": 37, \"character\": \"Shuttle Co-Pilot\", \"credit_id\": \"52fe48009251416c750aca6f\", \"gender\": 0, \"id\": 1180936, \"name\": \"Sean Patrick Murphy\", \"order\": 17}, {\"cast_id\": 38, \"character\": \"Shuttle Crew Chief\", \"credit_id\": \"52fe48009251416c750aca73\", \"gender\": 2, \"id\": 1019578, \"name\": \"Peter Dillon\", \"order\": 18}, {\"cast_id\": 39, \"character\": \"Tractor Operator / Troupe\", \"credit_id\": \"52fe48009251416c750aca77\", \"gender\": 0, \"id\": 91443, \"name\": \"Kevin Dorman\", \"order\": 19}, {\"cast_id\": 40, \"character\": \"Dragon Gunship Pilot\", \"credit_id\": \"52fe48009251416c750aca7b\", \"gender\": 2, \"id\": 173391, \"name\": \"Kelson Henderson\", \"order\": 20}, {\"cast_id\": 41, \"character\": \"Dragon Gunship Gunner\", \"credit_id\": \"52fe48009251416c750aca7f\", \"gender\": 0, \"id\": 1207236, \"name\": \"David Van Horn\", \"order\": 21}, {\"cast_id\": 42, \"character\": \"Dragon Gunship Navigator\", \"credit_id\": \"52fe48009251416c750aca83\", \"gender\": 0, \"id\": 215913, \"name\": \"Jacob Tomuri\", \"order\": 22}, {\"cast_id\": 43, \"character\": \"Suit #1\", \"credit_id\": \"52fe48009251416c750aca87\", \"gender\": 0, \"id\": 143206, \"name\": \"Michael Blain-Rozgay\", \"order\": 23}, {\"cast_id\": 44, \"character\": \"Suit #2\", \"credit_id\": \"52fe48009251416c750aca8b\", \"gender\": 2, \"id\": 169676, \"name\": \"Jon Curry\", \"order\": 24}, {\"cast_id\": 46, \"character\": \"Ambient Room Tech\", \"credit_id\": \"52fe48009251416c750aca8f\", \"gender\": 0, \"id\": 1048610, \"name\": \"Luke Hawker\", \"order\": 25}, {\"cast_id\": 47, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48009251416c750aca93\", \"gender\": 0, \"id\": 42288, \"name\": \"Woody Schultz\", \"order\": 26}, {\"cast_id\": 48, \"character\": \"Horse Clan Leader\", \"credit_id\": \"52fe48009251416c750aca97\", \"gender\": 2, \"id\": 68278, \"name\": \"Peter Mensah\", \"order\": 27}, {\"cast_id\": 49, \"character\": \"Link Room Tech\", \"credit_id\": \"52fe48009251416c750aca9b\", \"gender\": 0, \"id\": 1207247, \"name\": \"Sonia Yee\", \"order\": 28}, {\"cast_id\": 50, \"character\": \"Basketball Avatar / Troupe\", \"credit_id\": \"52fe48009251416c750aca9f\", \"gender\": 1, \"id\": 1207248, \"name\": \"Jahnel Curfman\", \"order\": 29}, {\"cast_id\": 51, \"character\": \"Basketball Avatar\", \"credit_id\": \"52fe48009251416c750acaa3\", \"gender\": 0, \"id\": 89714, \"name\": \"Ilram Choi\", \"order\": 30}, {\"cast_id\": 52, \"character\": \"Na\\'vi Child\", \"credit_id\": \"52fe48009251416c750acaa7\", \"gender\": 0, \"id\": 1207249, \"name\": \"Kyla Warren\", \"order\": 31}, {\"cast_id\": 53, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaab\", \"gender\": 0, \"id\": 1207250, \"name\": \"Lisa Roumain\", \"order\": 32}, {\"cast_id\": 54, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acaaf\", \"gender\": 1, \"id\": 83105, \"name\": \"Debra Wilson\", \"order\": 33}, {\"cast_id\": 57, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabb\", \"gender\": 0, \"id\": 1207253, \"name\": \"Chris Mala\", \"order\": 34}, {\"cast_id\": 55, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab3\", \"gender\": 0, \"id\": 1207251, \"name\": \"Taylor Kibby\", \"order\": 35}, {\"cast_id\": 56, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acab7\", \"gender\": 0, \"id\": 1207252, \"name\": \"Jodie Landau\", \"order\": 36}, {\"cast_id\": 58, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acabf\", \"gender\": 0, \"id\": 1207254, \"name\": \"Julie Lamm\", \"order\": 37}, {\"cast_id\": 59, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac3\", \"gender\": 0, \"id\": 1207257, \"name\": \"Cullen B. Madden\", \"order\": 38}, {\"cast_id\": 60, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acac7\", \"gender\": 0, \"id\": 1207259, \"name\": \"Joseph Brady Madden\", \"order\": 39}, {\"cast_id\": 61, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacb\", \"gender\": 0, \"id\": 1207262, \"name\": \"Frankie Torres\", \"order\": 40}, {\"cast_id\": 62, \"character\": \"Troupe\", \"credit_id\": \"52fe48009251416c750acacf\", \"gender\": 1, \"id\": 1158600, \"name\": \"Austin Wilson\", \"order\": 41}, {\"cast_id\": 63, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad3\", \"gender\": 1, \"id\": 983705, \"name\": \"Sara Wilson\", \"order\": 42}, {\"cast_id\": 64, \"character\": \"Troupe\", \"credit_id\": \"52fe48019251416c750acad7\", \"gender\": 0, \"id\": 1207263, \"name\": \"Tamica Washington-Miller\", \"order\": 43}, {\"cast_id\": 65, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadb\", \"gender\": 1, \"id\": 1145098, \"name\": \"Lucy Briant\", \"order\": 44}, {\"cast_id\": 66, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acadf\", \"gender\": 2, \"id\": 33305, \"name\": \"Nathan Meister\", \"order\": 45}, {\"cast_id\": 67, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae3\", \"gender\": 0, \"id\": 1207264, \"name\": \"Gerry Blair\", \"order\": 46}, {\"cast_id\": 68, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acae7\", \"gender\": 2, \"id\": 33311, \"name\": \"Matthew Chamberlain\", \"order\": 47}, {\"cast_id\": 69, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaeb\", \"gender\": 0, \"id\": 1207265, \"name\": \"Paul Yates\", \"order\": 48}, {\"cast_id\": 70, \"character\": \"Op Center Duty Officer\", \"credit_id\": \"52fe48019251416c750acaef\", \"gender\": 0, \"id\": 1207266, \"name\": \"Wray Wilson\", \"order\": 49}, {\"cast_id\": 71, \"character\": \"Op Center Staff\", \"credit_id\": \"52fe48019251416c750acaf3\", \"gender\": 2, \"id\": 54492, \"name\": \"James Gaylyn\", \"order\": 50}, {\"cast_id\": 72, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaf7\", \"gender\": 0, \"id\": 1207267, \"name\": \"Melvin Leno Clark III\", \"order\": 51}, {\"cast_id\": 73, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acafb\", \"gender\": 0, \"id\": 1207268, \"name\": \"Carvon Futrell\", \"order\": 52}, {\"cast_id\": 74, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acaff\", \"gender\": 0, \"id\": 1207269, \"name\": \"Brandon Jelkes\", \"order\": 53}, {\"cast_id\": 75, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb03\", \"gender\": 0, \"id\": 1207270, \"name\": \"Micah Moch\", \"order\": 54}, {\"cast_id\": 76, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb07\", \"gender\": 0, \"id\": 1207271, \"name\": \"Hanniyah Muhammad\", \"order\": 55}, {\"cast_id\": 77, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0b\", \"gender\": 0, \"id\": 1207272, \"name\": \"Christopher Nolen\", \"order\": 56}, {\"cast_id\": 78, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb0f\", \"gender\": 0, \"id\": 1207273, \"name\": \"Christa Oliver\", \"order\": 57}, {\"cast_id\": 79, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb13\", \"gender\": 0, \"id\": 1207274, \"name\": \"April Marie Thomas\", \"order\": 58}, {\"cast_id\": 80, \"character\": \"Dancer\", \"credit_id\": \"52fe48019251416c750acb17\", \"gender\": 0, \"id\": 1207275, \"name\": \"Bravita A. Threatt\", \"order\": 59}, {\"cast_id\": 81, \"character\": \"Mining Chief (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1b\", \"gender\": 0, \"id\": 1207276, \"name\": \"Colin Bleasdale\", \"order\": 60}, {\"cast_id\": 82, \"character\": \"Veteran Miner (uncredited)\", \"credit_id\": \"52fe48019251416c750acb1f\", \"gender\": 0, \"id\": 107969, \"name\": \"Mike Bodnar\", \"order\": 61}, {\"cast_id\": 83, \"character\": \"Richard (uncredited)\", \"credit_id\": \"52fe48019251416c750acb23\", \"gender\": 0, \"id\": 1207278, \"name\": \"Matt Clayton\", \"order\": 62}, {\"cast_id\": 84, \"character\": \"Nav\\'i (uncredited)\", \"credit_id\": \"52fe48019251416c750acb27\", \"gender\": 1, \"id\": 147898, \"name\": \"Nicole Dionne\", \"order\": 63}, {\"cast_id\": 85, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2b\", \"gender\": 0, \"id\": 1207280, \"name\": \"Jamie Harrison\", \"order\": 64}, {\"cast_id\": 86, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb2f\", \"gender\": 0, \"id\": 1207281, \"name\": \"Allan Henry\", \"order\": 65}, {\"cast_id\": 87, \"character\": \"Ground Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb33\", \"gender\": 2, \"id\": 1207282, \"name\": \"Anthony Ingruber\", \"order\": 66}, {\"cast_id\": 88, \"character\": \"Flight Crew Mechanic (uncredited)\", \"credit_id\": \"52fe48019251416c750acb37\", \"gender\": 0, \"id\": 1207283, \"name\": \"Ashley Jeffery\", \"order\": 67}, {\"cast_id\": 14, \"character\": \"Samson Pilot\", \"credit_id\": \"52fe48009251416c750ac9f9\", \"gender\": 0, \"id\": 98216, \"name\": \"Dean Knowsley\", \"order\": 68}, {\"cast_id\": 89, \"character\": \"Trooper (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3b\", \"gender\": 0, \"id\": 1201399, \"name\": \"Joseph Mika-Hunt\", \"order\": 69}, {\"cast_id\": 90, \"character\": \"Banshee (uncredited)\", \"credit_id\": \"52fe48019251416c750acb3f\", \"gender\": 0, \"id\": 236696, \"name\": \"Terry Notary\", \"order\": 70}, {\"cast_id\": 91, \"character\": \"Soldier (uncredited)\", \"credit_id\": \"52fe48019251416c750acb43\", \"gender\": 0, \"id\": 1207287, \"name\": \"Kai Pantano\", \"order\": 71}, {\"cast_id\": 92, \"character\": \"Blast Technician (uncredited)\", \"credit_id\": \"52fe48019251416c750acb47\", \"gender\": 0, \"id\": 1207288, \"name\": \"Logan Pithyou\", \"order\": 72}, {\"cast_id\": 93, \"character\": \"Vindum Raah (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4b\", \"gender\": 0, \"id\": 1207289, \"name\": \"Stuart Pollock\", \"order\": 73}, {\"cast_id\": 94, \"character\": \"Hero (uncredited)\", \"credit_id\": \"52fe48019251416c750acb4f\", \"gender\": 0, \"id\": 584868, \"name\": \"Raja\", \"order\": 74}, {\"cast_id\": 95, \"character\": \"Ops Centreworker (uncredited)\", \"credit_id\": \"52fe48019251416c750acb53\", \"gender\": 0, \"id\": 1207290, \"name\": \"Gareth Ruck\", \"order\": 75}, {\"cast_id\": 96, \"character\": \"Engineer (uncredited)\", \"credit_id\": \"52fe48019251416c750acb57\", \"gender\": 0, \"id\": 1062463, \"name\": \"Rhian Sheehan\", \"order\": 76}, {\"cast_id\": 97, \"character\": \"Col. Quaritch\\'s Mech Suit (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5b\", \"gender\": 0, \"id\": 60656, \"name\": \"T. J. Storm\", \"order\": 77}, {\"cast_id\": 98, \"character\": \"Female Marine (uncredited)\", \"credit_id\": \"52fe48019251416c750acb5f\", \"gender\": 0, \"id\": 1207291, \"name\": \"Jodie Taylor\", \"order\": 78}, {\"cast_id\": 99, \"character\": \"Ikran Clan Leader (uncredited)\", \"credit_id\": \"52fe48019251416c750acb63\", \"gender\": 1, \"id\": 1186027, \"name\": \"Alicia Vela-Bailey\", \"order\": 79}, {\"cast_id\": 100, \"character\": \"Geologist (uncredited)\", \"credit_id\": \"52fe48019251416c750acb67\", \"gender\": 0, \"id\": 1207292, \"name\": \"Richard Whiteside\", \"order\": 80}, {\"cast_id\": 101, \"character\": \"Na\\'vi (uncredited)\", \"credit_id\": \"52fe48019251416c750acb6b\", \"gender\": 0, \"id\": 103259, \"name\": \"Nikie Zambo\", \"order\": 81}, {\"cast_id\": 102, \"character\": \"Ambient Room Tech / Troupe\", \"credit_id\": \"52fe48019251416c750acb6f\", \"gender\": 1, \"id\": 42286, \"name\": \"Julene Renee\", \"order\": 82}]'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.cast[0] # We will include the \"name\" of first 4(you can vary) cast members"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "9290b274",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sam Worthington Zoe Saldana Sigourney Weaver Stephen Lang'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join([i['name'] for i in eval(df.cast[0])[:4]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "3d8917a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"credit_id\": \"52fe48009251416c750aca23\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1721, \"job\": \"Editor\", \"name\": \"Stephen E. Rivkin\"}, {\"credit_id\": \"539c47ecc3a36810e3001f87\", \"department\": \"Art\", \"gender\": 2, \"id\": 496, \"job\": \"Production Design\", \"name\": \"Rick Carter\"}, {\"credit_id\": \"54491c89c3a3680fb4001cf7\", \"department\": \"Sound\", \"gender\": 0, \"id\": 900, \"job\": \"Sound Designer\", \"name\": \"Christopher Boyes\"}, {\"credit_id\": \"54491cb70e0a267480001bd0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 900, \"job\": \"Supervising Sound Editor\", \"name\": \"Christopher Boyes\"}, {\"credit_id\": \"539c4a4cc3a36810c9002101\", \"department\": \"Production\", \"gender\": 1, \"id\": 1262, \"job\": \"Casting\", \"name\": \"Mali Finn\"}, {\"credit_id\": \"5544ee3b925141499f0008fc\", \"department\": \"Sound\", \"gender\": 2, \"id\": 1729, \"job\": \"Original Music Composer\", \"name\": \"James Horner\"}, {\"credit_id\": \"52fe48009251416c750ac9c3\", \"department\": \"Directing\", \"gender\": 2, \"id\": 2710, \"job\": \"Director\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750ac9d9\", \"department\": \"Writing\", \"gender\": 2, \"id\": 2710, \"job\": \"Writer\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca17\", \"department\": \"Editing\", \"gender\": 2, \"id\": 2710, \"job\": \"Editor\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca29\", \"department\": \"Production\", \"gender\": 2, \"id\": 2710, \"job\": \"Producer\", \"name\": \"James Cameron\"}, {\"credit_id\": \"52fe48009251416c750aca3f\", \"department\": \"Writing\", \"gender\": 2, \"id\": 2710, \"job\": \"Screenplay\", \"name\": \"James Cameron\"}, {\"credit_id\": \"539c4987c3a36810ba0021a4\", \"department\": \"Art\", \"gender\": 2, \"id\": 7236, \"job\": \"Art Direction\", \"name\": \"Andrew Menzies\"}, {\"credit_id\": \"549598c3c3a3686ae9004383\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 6690, \"job\": \"Visual Effects Producer\", \"name\": \"Jill Brooks\"}, {\"credit_id\": \"52fe48009251416c750aca4b\", \"department\": \"Production\", \"gender\": 1, \"id\": 6347, \"job\": \"Casting\", \"name\": \"Margery Simkin\"}, {\"credit_id\": \"570b6f419251417da70032fe\", \"department\": \"Art\", \"gender\": 2, \"id\": 6878, \"job\": \"Supervising Art Director\", \"name\": \"Kevin Ishioka\"}, {\"credit_id\": \"5495a0fac3a3686ae9004468\", \"department\": \"Sound\", \"gender\": 0, \"id\": 6883, \"job\": \"Music Editor\", \"name\": \"Dick Bernstein\"}, {\"credit_id\": \"54959706c3a3686af3003e81\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8159, \"job\": \"Sound Effects Editor\", \"name\": \"Shannon Mills\"}, {\"credit_id\": \"54491d58c3a3680fb1001ccb\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8160, \"job\": \"Foley\", \"name\": \"Dennie Thorpe\"}, {\"credit_id\": \"54491d6cc3a3680fa5001b2c\", \"department\": \"Sound\", \"gender\": 0, \"id\": 8163, \"job\": \"Foley\", \"name\": \"Jana Vance\"}, {\"credit_id\": \"52fe48009251416c750aca57\", \"department\": \"Costume & Make-Up\", \"gender\": 1, \"id\": 8527, \"job\": \"Costume Design\", \"name\": \"Deborah Lynn Scott\"}, {\"credit_id\": \"52fe48009251416c750aca2f\", \"department\": \"Production\", \"gender\": 2, \"id\": 8529, \"job\": \"Producer\", \"name\": \"Jon Landau\"}, {\"credit_id\": \"539c4937c3a36810ba002194\", \"department\": \"Art\", \"gender\": 0, \"id\": 9618, \"job\": \"Art Direction\", \"name\": \"Sean Haworth\"}, {\"credit_id\": \"539c49b6c3a36810c10020e6\", \"department\": \"Art\", \"gender\": 1, \"id\": 12653, \"job\": \"Set Decoration\", \"name\": \"Kim Sinclair\"}, {\"credit_id\": \"570b6f2f9251413a0e00020d\", \"department\": \"Art\", \"gender\": 1, \"id\": 12653, \"job\": \"Supervising Art Director\", \"name\": \"Kim Sinclair\"}, {\"credit_id\": \"54491a6c0e0a26748c001b19\", \"department\": \"Art\", \"gender\": 2, \"id\": 14350, \"job\": \"Set Designer\", \"name\": \"Richard F. Mays\"}, {\"credit_id\": \"56928cf4c3a3684cff0025c4\", \"department\": \"Production\", \"gender\": 1, \"id\": 20294, \"job\": \"Executive Producer\", \"name\": \"Laeta Kalogridis\"}, {\"credit_id\": \"52fe48009251416c750aca51\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 17675, \"job\": \"Costume Design\", \"name\": \"Mayes C. Rubeo\"}, {\"credit_id\": \"52fe48009251416c750aca11\", \"department\": \"Camera\", \"gender\": 2, \"id\": 18265, \"job\": \"Director of Photography\", \"name\": \"Mauro Fiore\"}, {\"credit_id\": \"5449194d0e0a26748f001b39\", \"department\": \"Art\", \"gender\": 0, \"id\": 42281, \"job\": \"Set Designer\", \"name\": \"Scott Herbertson\"}, {\"credit_id\": \"52fe48009251416c750aca05\", \"department\": \"Crew\", \"gender\": 0, \"id\": 42288, \"job\": \"Stunts\", \"name\": \"Woody Schultz\"}, {\"credit_id\": \"5592aefb92514152de0010f5\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 29067, \"job\": \"Makeup Artist\", \"name\": \"Linda DeVetta\"}, {\"credit_id\": \"5592afa492514152de00112c\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 29067, \"job\": \"Hairstylist\", \"name\": \"Linda DeVetta\"}, {\"credit_id\": \"54959ed592514130fc002e5d\", \"department\": \"Camera\", \"gender\": 2, \"id\": 33302, \"job\": \"Camera Operator\", \"name\": \"Richard Bluck\"}, {\"credit_id\": \"539c4891c3a36810ba002147\", \"department\": \"Art\", \"gender\": 2, \"id\": 33303, \"job\": \"Art Direction\", \"name\": \"Simon Bright\"}, {\"credit_id\": \"54959c069251417a81001f3a\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 113145, \"job\": \"Visual Effects Supervisor\", \"name\": \"Richard Martin\"}, {\"credit_id\": \"54959a0dc3a3680ff5002c8d\", \"department\": \"Crew\", \"gender\": 2, \"id\": 58188, \"job\": \"Visual Effects Editor\", \"name\": \"Steve R. Moore\"}, {\"credit_id\": \"52fe48009251416c750aca1d\", \"department\": \"Editing\", \"gender\": 2, \"id\": 58871, \"job\": \"Editor\", \"name\": \"John Refoua\"}, {\"credit_id\": \"54491a4dc3a3680fc30018ca\", \"department\": \"Art\", \"gender\": 0, \"id\": 92359, \"job\": \"Set Designer\", \"name\": \"Karl J. Martin\"}, {\"credit_id\": \"52fe48009251416c750aca35\", \"department\": \"Camera\", \"gender\": 1, \"id\": 72201, \"job\": \"Director of Photography\", \"name\": \"Chiling Lin\"}, {\"credit_id\": \"52fe48009251416c750ac9ff\", \"department\": \"Crew\", \"gender\": 0, \"id\": 89714, \"job\": \"Stunts\", \"name\": \"Ilram Choi\"}, {\"credit_id\": \"54959c529251416e2b004394\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 93214, \"job\": \"Visual Effects Supervisor\", \"name\": \"Steven Quale\"}, {\"credit_id\": \"54491edf0e0a267489001c37\", \"department\": \"Crew\", \"gender\": 1, \"id\": 122607, \"job\": \"Dialect Coach\", \"name\": \"Carla Meyer\"}, {\"credit_id\": \"539c485bc3a368653d001a3a\", \"department\": \"Art\", \"gender\": 2, \"id\": 132585, \"job\": \"Art Direction\", \"name\": \"Nick Bassett\"}, {\"credit_id\": \"539c4903c3a368653d001a74\", \"department\": \"Art\", \"gender\": 0, \"id\": 132596, \"job\": \"Art Direction\", \"name\": \"Jill Cormack\"}, {\"credit_id\": \"539c4967c3a368653d001a94\", \"department\": \"Art\", \"gender\": 0, \"id\": 132604, \"job\": \"Art Direction\", \"name\": \"Andy McLaren\"}, {\"credit_id\": \"52fe48009251416c750aca45\", \"department\": \"Crew\", \"gender\": 0, \"id\": 236696, \"job\": \"Motion Capture Artist\", \"name\": \"Terry Notary\"}, {\"credit_id\": \"54959e02c3a3680fc60027d2\", \"department\": \"Crew\", \"gender\": 2, \"id\": 956198, \"job\": \"Stunt Coordinator\", \"name\": \"Garrett Warren\"}, {\"credit_id\": \"54959ca3c3a3686ae300438c\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 957874, \"job\": \"Visual Effects Supervisor\", \"name\": \"Jonathan Rothbart\"}, {\"credit_id\": \"570b6f519251412c74001b2f\", \"department\": \"Art\", \"gender\": 0, \"id\": 957889, \"job\": \"Supervising Art Director\", \"name\": \"Stefan Dechant\"}, {\"credit_id\": \"570b6f62c3a3680b77007460\", \"department\": \"Art\", \"gender\": 2, \"id\": 959555, \"job\": \"Supervising Art Director\", \"name\": \"Todd Cherniawsky\"}, {\"credit_id\": \"539c4a3ac3a36810da0021cc\", \"department\": \"Production\", \"gender\": 0, \"id\": 1016177, \"job\": \"Casting\", \"name\": \"Miranda Rivers\"}, {\"credit_id\": \"539c482cc3a36810c1002062\", \"department\": \"Art\", \"gender\": 0, \"id\": 1032536, \"job\": \"Production Design\", \"name\": \"Robert Stromberg\"}, {\"credit_id\": \"539c4b65c3a36810c9002125\", \"department\": \"Costume & Make-Up\", \"gender\": 2, \"id\": 1071680, \"job\": \"Costume Design\", \"name\": \"John Harding\"}, {\"credit_id\": \"54959e6692514130fc002e4e\", \"department\": \"Camera\", \"gender\": 0, \"id\": 1177364, \"job\": \"Steadicam Operator\", \"name\": \"Roberto De Angelis\"}, {\"credit_id\": \"539c49f1c3a368653d001aac\", \"department\": \"Costume & Make-Up\", \"gender\": 2, \"id\": 1202850, \"job\": \"Makeup Department Head\", \"name\": \"Mike Smithson\"}, {\"credit_id\": \"5495999ec3a3686ae100460c\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1204668, \"job\": \"Visual Effects Producer\", \"name\": \"Alain Lalanne\"}, {\"credit_id\": \"54959cdfc3a3681153002729\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1206410, \"job\": \"Visual Effects Supervisor\", \"name\": \"Lucas Salton\"}, {\"credit_id\": \"549596239251417a81001eae\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1234266, \"job\": \"Post Production Supervisor\", \"name\": \"Janace Tashjian\"}, {\"credit_id\": \"54959c859251416e1e003efe\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1271932, \"job\": \"Visual Effects Supervisor\", \"name\": \"Stephen Rosenbaum\"}, {\"credit_id\": \"5592af28c3a368775a00105f\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1310064, \"job\": \"Makeup Artist\", \"name\": \"Frankie Karena\"}, {\"credit_id\": \"539c4adfc3a36810e300203b\", \"department\": \"Costume & Make-Up\", \"gender\": 1, \"id\": 1319844, \"job\": \"Costume Supervisor\", \"name\": \"Lisa Lovaas\"}, {\"credit_id\": \"54959b579251416e2b004371\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1327028, \"job\": \"Visual Effects Supervisor\", \"name\": \"Jonathan Fawkner\"}, {\"credit_id\": \"539c48a7c3a36810b5001fa7\", \"department\": \"Art\", \"gender\": 0, \"id\": 1330561, \"job\": \"Art Direction\", \"name\": \"Robert Bavin\"}, {\"credit_id\": \"539c4a71c3a36810da0021e0\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330567, \"job\": \"Costume Supervisor\", \"name\": \"Anthony Almaraz\"}, {\"credit_id\": \"539c4a8ac3a36810ba0021e4\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330570, \"job\": \"Costume Supervisor\", \"name\": \"Carolyn M. Fenton\"}, {\"credit_id\": \"539c4ab6c3a36810da0021f0\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1330574, \"job\": \"Costume Supervisor\", \"name\": \"Beth Koenigsberg\"}, {\"credit_id\": \"54491ab70e0a267480001ba2\", \"department\": \"Art\", \"gender\": 0, \"id\": 1336191, \"job\": \"Set Designer\", \"name\": \"Sam Page\"}, {\"credit_id\": \"544919d9c3a3680fc30018bd\", \"department\": \"Art\", \"gender\": 0, \"id\": 1339441, \"job\": \"Set Designer\", \"name\": \"Tex Kadonaga\"}, {\"credit_id\": \"54491cf50e0a267483001b0c\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1352422, \"job\": \"Dialogue Editor\", \"name\": \"Kim Foscato\"}, {\"credit_id\": \"544919f40e0a26748c001b09\", \"department\": \"Art\", \"gender\": 0, \"id\": 1352962, \"job\": \"Set Designer\", \"name\": \"Tammy S. Lee\"}, {\"credit_id\": \"5495a115c3a3680ff5002d71\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1357070, \"job\": \"Transportation Coordinator\", \"name\": \"Denny Caira\"}, {\"credit_id\": \"5495a12f92514130fc002e94\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1357071, \"job\": \"Transportation Coordinator\", \"name\": \"James Waitkus\"}, {\"credit_id\": \"5495976fc3a36811530026b0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1360103, \"job\": \"Supervising Sound Editor\", \"name\": \"Addison Teague\"}, {\"credit_id\": \"54491837c3a3680fb1001c5a\", \"department\": \"Art\", \"gender\": 2, \"id\": 1376887, \"job\": \"Set Designer\", \"name\": \"C. Scott Baker\"}, {\"credit_id\": \"54491878c3a3680fb4001c9d\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376888, \"job\": \"Set Designer\", \"name\": \"Luke Caska\"}, {\"credit_id\": \"544918dac3a3680fa5001ae0\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376889, \"job\": \"Set Designer\", \"name\": \"David Chow\"}, {\"credit_id\": \"544919110e0a267486001b68\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376890, \"job\": \"Set Designer\", \"name\": \"Jonathan Dyer\"}, {\"credit_id\": \"54491967c3a3680faa001b5e\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376891, \"job\": \"Set Designer\", \"name\": \"Joseph Hiura\"}, {\"credit_id\": \"54491997c3a3680fb1001c8a\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376892, \"job\": \"Art Department Coordinator\", \"name\": \"Rebecca Jellie\"}, {\"credit_id\": \"544919ba0e0a26748f001b42\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376893, \"job\": \"Set Designer\", \"name\": \"Robert Andrew Johnson\"}, {\"credit_id\": \"54491b1dc3a3680faa001b8c\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376895, \"job\": \"Assistant Art Director\", \"name\": \"Mike Stassi\"}, {\"credit_id\": \"54491b79c3a3680fbb001826\", \"department\": \"Art\", \"gender\": 0, \"id\": 1376897, \"job\": \"Construction Coordinator\", \"name\": \"John Villarino\"}, {\"credit_id\": \"54491baec3a3680fb4001ce6\", \"department\": \"Art\", \"gender\": 2, \"id\": 1376898, \"job\": \"Assistant Art Director\", \"name\": \"Jeffrey Wisniewski\"}, {\"credit_id\": \"54491d2fc3a3680fb4001d07\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1376899, \"job\": \"Dialogue Editor\", \"name\": \"Cheryl Nardi\"}, {\"credit_id\": \"54491d86c3a3680fa5001b2f\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1376901, \"job\": \"Dialogue Editor\", \"name\": \"Marshall Winn\"}, {\"credit_id\": \"54491d9dc3a3680faa001bb0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1376902, \"job\": \"Supervising Sound Editor\", \"name\": \"Gwendolyn Yates Whittle\"}, {\"credit_id\": \"54491dc10e0a267486001bce\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1376903, \"job\": \"Sound Re-Recording Mixer\", \"name\": \"William Stein\"}, {\"credit_id\": \"54491f500e0a26747c001c07\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1376909, \"job\": \"Choreographer\", \"name\": \"Lula Washington\"}, {\"credit_id\": \"549599239251412c4e002a2e\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1391692, \"job\": \"Visual Effects Producer\", \"name\": \"Chris Del Conte\"}, {\"credit_id\": \"54959d54c3a36831b8001d9a\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 1391695, \"job\": \"Visual Effects Supervisor\", \"name\": \"R. Christopher White\"}, {\"credit_id\": \"54959bdf9251412c4e002a66\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1394070, \"job\": \"Visual Effects Supervisor\", \"name\": \"Dan Lemmon\"}, {\"credit_id\": \"5495971d92514132ed002922\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1394129, \"job\": \"Sound Effects Editor\", \"name\": \"Tim Nielsen\"}, {\"credit_id\": \"5592b25792514152cc0011aa\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1394286, \"job\": \"CG Supervisor\", \"name\": \"Michael Mulholland\"}, {\"credit_id\": \"54959a329251416e2b004355\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1394750, \"job\": \"Visual Effects Editor\", \"name\": \"Thomas Nittmann\"}, {\"credit_id\": \"54959d6dc3a3686ae9004401\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1394755, \"job\": \"Visual Effects Supervisor\", \"name\": \"Edson Williams\"}, {\"credit_id\": \"5495a08fc3a3686ae300441c\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1394953, \"job\": \"Digital Intermediate\", \"name\": \"Christine Carr\"}, {\"credit_id\": \"55402d659251413d6d000249\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1395269, \"job\": \"Visual Effects Supervisor\", \"name\": \"John Bruno\"}, {\"credit_id\": \"54959e7b9251416e1e003f3e\", \"department\": \"Camera\", \"gender\": 0, \"id\": 1398970, \"job\": \"Steadicam Operator\", \"name\": \"David Emmerichs\"}, {\"credit_id\": \"54959734c3a3686ae10045e0\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1400906, \"job\": \"Sound Effects Editor\", \"name\": \"Christopher Scarabosio\"}, {\"credit_id\": \"549595dd92514130fc002d79\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401784, \"job\": \"Production Supervisor\", \"name\": \"Jennifer Teves\"}, {\"credit_id\": \"549596009251413af70028cc\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401785, \"job\": \"Production Manager\", \"name\": \"Brigitte Yorke\"}, {\"credit_id\": \"549596e892514130fc002d99\", \"department\": \"Sound\", \"gender\": 0, \"id\": 1401786, \"job\": \"Sound Effects Editor\", \"name\": \"Ken Fischer\"}, {\"credit_id\": \"549598229251412c4e002a1c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401787, \"job\": \"Special Effects Coordinator\", \"name\": \"Iain Hutton\"}, {\"credit_id\": \"549598349251416e2b00432b\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401788, \"job\": \"Special Effects Coordinator\", \"name\": \"Steve Ingram\"}, {\"credit_id\": \"54959905c3a3686ae3004324\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401789, \"job\": \"Visual Effects Producer\", \"name\": \"Joyce Cox\"}, {\"credit_id\": \"5495994b92514132ed002951\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401790, \"job\": \"Visual Effects Producer\", \"name\": \"Jenny Foster\"}, {\"credit_id\": \"549599cbc3a3686ae1004613\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401791, \"job\": \"Visual Effects Editor\", \"name\": \"Christopher Marino\"}, {\"credit_id\": \"549599f2c3a3686ae100461e\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401792, \"job\": \"Visual Effects Editor\", \"name\": \"Jim Milton\"}, {\"credit_id\": \"54959a51c3a3686af3003eb5\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401793, \"job\": \"Visual Effects Producer\", \"name\": \"Cyndi Ochs\"}, {\"credit_id\": \"54959a7cc3a36811530026f4\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401794, \"job\": \"Visual Effects Editor\", \"name\": \"Lucas Putnam\"}, {\"credit_id\": \"54959b91c3a3680ff5002cb4\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401795, \"job\": \"Visual Effects Supervisor\", \"name\": \"Anthony \\'Max\\' Ivins\"}, {\"credit_id\": \"54959bb69251412c4e002a5f\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401796, \"job\": \"Visual Effects Supervisor\", \"name\": \"John Knoll\"}, {\"credit_id\": \"54959cbbc3a3686ae3004391\", \"department\": \"Visual Effects\", \"gender\": 2, \"id\": 1401799, \"job\": \"Visual Effects Supervisor\", \"name\": \"Eric Saindon\"}, {\"credit_id\": \"54959d06c3a3686ae90043f6\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401800, \"job\": \"Visual Effects Supervisor\", \"name\": \"Wayne Stables\"}, {\"credit_id\": \"54959d259251416e1e003f11\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401801, \"job\": \"Visual Effects Supervisor\", \"name\": \"David Stinnett\"}, {\"credit_id\": \"54959db49251413af7002975\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401803, \"job\": \"Visual Effects Supervisor\", \"name\": \"Guy Williams\"}, {\"credit_id\": \"54959de4c3a3681153002750\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1401804, \"job\": \"Stunt Coordinator\", \"name\": \"Stuart Thorp\"}, {\"credit_id\": \"54959ef2c3a3680fc60027f2\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1401805, \"job\": \"Best Boy Electric\", \"name\": \"Giles Coburn\"}, {\"credit_id\": \"54959f07c3a3680fc60027f9\", \"department\": \"Camera\", \"gender\": 2, \"id\": 1401806, \"job\": \"Still Photographer\", \"name\": \"Mark Fellman\"}, {\"credit_id\": \"54959f47c3a3681153002774\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1401807, \"job\": \"Lighting Technician\", \"name\": \"Scott Sprague\"}, {\"credit_id\": \"54959f8cc3a36831b8001df2\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401808, \"job\": \"Animation Director\", \"name\": \"Jeremy Hollobon\"}, {\"credit_id\": \"54959fa0c3a36831b8001dfb\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401809, \"job\": \"Animation Director\", \"name\": \"Orlando Meunier\"}, {\"credit_id\": \"54959fb6c3a3686af3003f54\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1401810, \"job\": \"Animation Director\", \"name\": \"Taisuke Tanimura\"}, {\"credit_id\": \"54959fd2c3a36831b8001e02\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1401812, \"job\": \"Set Costumer\", \"name\": \"Lilia Mishel Acevedo\"}, {\"credit_id\": \"54959ff9c3a3686ae300440c\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1401814, \"job\": \"Set Costumer\", \"name\": \"Alejandro M. Hernandez\"}, {\"credit_id\": \"5495a0ddc3a3686ae10046fe\", \"department\": \"Editing\", \"gender\": 0, \"id\": 1401815, \"job\": \"Digital Intermediate\", \"name\": \"Marvin Hall\"}, {\"credit_id\": \"5495a1f7c3a3686ae3004443\", \"department\": \"Production\", \"gender\": 0, \"id\": 1401816, \"job\": \"Publicist\", \"name\": \"Judy Alley\"}, {\"credit_id\": \"5592b29fc3a36869d100002f\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1418381, \"job\": \"CG Supervisor\", \"name\": \"Mike Perry\"}, {\"credit_id\": \"5592b23a9251415df8001081\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1426854, \"job\": \"CG Supervisor\", \"name\": \"Andrew Morley\"}, {\"credit_id\": \"55491e1192514104c40002d8\", \"department\": \"Art\", \"gender\": 0, \"id\": 1438901, \"job\": \"Conceptual Design\", \"name\": \"Seth Engstrom\"}, {\"credit_id\": \"5525d5809251417276002b06\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1447362, \"job\": \"Visual Effects Art Director\", \"name\": \"Eric Oliver\"}, {\"credit_id\": \"554427ca925141586500312a\", \"department\": \"Visual Effects\", \"gender\": 0, \"id\": 1447503, \"job\": \"Modeling\", \"name\": \"Matsune Suzuki\"}, {\"credit_id\": \"551906889251415aab001c88\", \"department\": \"Art\", \"gender\": 0, \"id\": 1447524, \"job\": \"Art Department Manager\", \"name\": \"Paul Tobin\"}, {\"credit_id\": \"5592af8492514152cc0010de\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1452643, \"job\": \"Hairstylist\", \"name\": \"Roxane Griffin\"}, {\"credit_id\": \"553d3c109251415852001318\", \"department\": \"Lighting\", \"gender\": 0, \"id\": 1453938, \"job\": \"Lighting Artist\", \"name\": \"Arun Ram-Mohan\"}, {\"credit_id\": \"5592af4692514152d5001355\", \"department\": \"Costume & Make-Up\", \"gender\": 0, \"id\": 1457305, \"job\": \"Makeup Artist\", \"name\": \"Georgia Lockhart-Adams\"}, {\"credit_id\": \"5592b2eac3a36877470012a5\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1466035, \"job\": \"CG Supervisor\", \"name\": \"Thrain Shadbolt\"}, {\"credit_id\": \"5592b032c3a36877450015f1\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483220, \"job\": \"CG Supervisor\", \"name\": \"Brad Alexander\"}, {\"credit_id\": \"5592b05592514152d80012f6\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483221, \"job\": \"CG Supervisor\", \"name\": \"Shadi Almassizadeh\"}, {\"credit_id\": \"5592b090c3a36877570010b5\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483222, \"job\": \"CG Supervisor\", \"name\": \"Simon Clutterbuck\"}, {\"credit_id\": \"5592b0dbc3a368774b00112c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483223, \"job\": \"CG Supervisor\", \"name\": \"Graeme Demmocks\"}, {\"credit_id\": \"5592b0fe92514152db0010c1\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483224, \"job\": \"CG Supervisor\", \"name\": \"Adrian Fernandes\"}, {\"credit_id\": \"5592b11f9251415df8001059\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483225, \"job\": \"CG Supervisor\", \"name\": \"Mitch Gates\"}, {\"credit_id\": \"5592b15dc3a3687745001645\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483226, \"job\": \"CG Supervisor\", \"name\": \"Jerry Kung\"}, {\"credit_id\": \"5592b18e925141645a0004ae\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483227, \"job\": \"CG Supervisor\", \"name\": \"Andy Lomas\"}, {\"credit_id\": \"5592b1bfc3a368775d0010e7\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483228, \"job\": \"CG Supervisor\", \"name\": \"Sebastian Marino\"}, {\"credit_id\": \"5592b2049251415df8001078\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483229, \"job\": \"CG Supervisor\", \"name\": \"Matthias Menz\"}, {\"credit_id\": \"5592b27b92514152d800136a\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483230, \"job\": \"CG Supervisor\", \"name\": \"Sergei Nevshupov\"}, {\"credit_id\": \"5592b2c3c3a36869e800003c\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483231, \"job\": \"CG Supervisor\", \"name\": \"Philippe Rebours\"}, {\"credit_id\": \"5592b317c3a36877470012af\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483232, \"job\": \"CG Supervisor\", \"name\": \"Michael Takarangi\"}, {\"credit_id\": \"5592b345c3a36877470012bb\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483233, \"job\": \"CG Supervisor\", \"name\": \"David Weitzberg\"}, {\"credit_id\": \"5592b37cc3a368775100113b\", \"department\": \"Crew\", \"gender\": 0, \"id\": 1483234, \"job\": \"CG Supervisor\", \"name\": \"Ben White\"}, {\"credit_id\": \"573c8e2f9251413f5d000094\", \"department\": \"Crew\", \"gender\": 1, \"id\": 1621932, \"job\": \"Stunts\", \"name\": \"Min Windle\"}]'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.crew[0]  #We will consider the \"name\" tag of the \"job\" : \"Director\" and \"Producer\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "dd2f0135",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Jon Landau James Cameron'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "' '.join(list(set([i['name'] for i in eval(df.crew[0]) if (i['job'] == 'Director' or i['job'] == 'Producer')])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5c7950e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to fetch the desired details from the columns\n",
    "def get_required_details(overview, genres, cast, crew):\n",
    "    corpus = \"\"\n",
    "    genre = ' '.join([i['name'] for i in eval(genres)])\n",
    "    cast =  ' '.join([i['name'] for i in eval(cast)[:4]])\n",
    "    crew = ' '.join(list(set([i['name'] for i in eval(crew) if i['job']=='Director' or i['job']=='Producer'])))\n",
    "    corpus += overview+ \" \" + genre + \" \" + cast + \" \" + crew\n",
    "    return corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f7d069ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = []\n",
    "for i in range(len(df)):\n",
    "    corpus.append(get_required_details(df.iloc[i].overview, df.iloc[i].genres, df.iloc[i].cast, df.iloc[i].crew))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "167d9a2e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4800"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "9f3a904e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In the 22nd century, a paraplegic Marine is dispatched to the moon Pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization. Action Adventure Fantasy Science Fiction Sam Worthington Zoe Saldana Sigourney Weaver Stephen Lang Jon Landau James Cameron'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90b0552f",
   "metadata": {},
   "source": [
    "At this point, we have the option to add this newly created column to our DataFrame and either retain or drop the original columns — this choice is yours to make. In my case, I will opt to drop these columns to streamline and reduce the size of our DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "ca5115f5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\CHETAN\\AppData\\Local\\Temp\\ipykernel_508\\735948131.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df.rename(columns = {'title_x': 'title'}, inplace= True)\n"
     ]
    }
   ],
   "source": [
    "# rename title column\n",
    "df.rename(columns = {'title_x': 'title'}, inplace= True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "e3f3f4e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\CHETAN\\AppData\\Local\\Temp\\ipykernel_508\\3970332720.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df.drop(columns=['genres', 'overview', 'cast', 'crew'], inplace=True)\n"
     ]
    }
   ],
   "source": [
    "# drop old columns\n",
    "df.drop(columns=['genres', 'overview', 'cast', 'crew'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "5aa30de1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\CHETAN\\AppData\\Local\\Temp\\ipykernel_508\\2533634386.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  df['corpus'] = corpus\n"
     ]
    }
   ],
   "source": [
    "# add corpus\n",
    "df['corpus'] = corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1701a319",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>corpus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>6.1</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                     title  vote_average  \\\n",
       "0   19995                                    Avatar           7.2   \n",
       "1     285  Pirates of the Caribbean: At World's End           6.9   \n",
       "2  206647                                   Spectre           6.3   \n",
       "3   49026                     The Dark Knight Rises           7.6   \n",
       "4   49529                               John Carter           6.1   \n",
       "\n",
       "                                              corpus  \n",
       "0  In the 22nd century, a paraplegic Marine is di...  \n",
       "1  Captain Barbossa, long believed to be dead, ha...  \n",
       "2  A cryptic message from Bond’s past sends him o...  \n",
       "3  Following the death of District Attorney Harve...  \n",
       "4  John Carter is a war-weary, former military ca...  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c6315a4",
   "metadata": {},
   "source": [
    "## Text Representation and Text Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b92ead",
   "metadata": {},
   "source": [
    "To enable machine learning algorithms to process text data effectively, the textual information must undergo a transformation into a mathematical form, typically represented as vectors of numbers a.k.a **vector space model (VSM)**, where text units are encoded as numerical vectors.\n",
    "\n",
    "The task of establishing relationships between texts involves a two-step process. \n",
    "    The first step, as we previously discussed, involves **text representation, wherein the vectors are generated**. \n",
    "    Once we have these vectors, the second step is to **determine methods for comparing them to measure their similarities or differences**.\n",
    "    \n",
    "\n",
    "In the NLP universe, this conversion of raw text to a suitable numerical form is called **text representation**.\n",
    "Various techniques are used to collect texts split them into atoms, and transform them into numerical vectors.\n",
    "\n",
    "In order to correctly extract the main words/tokens from a piece of the corpus, the very first step usually is to **break the sentence into lexical units** then **tokenize the sentences** and do **removal of stopwords**, and irrelevant words/sentences depending upon the use case, which is usually done by either **regular expressions, NLTK, spaCy or TextVectorivectorization.**\n",
    "\n",
    "Following this, we will now have a **list of tokens that are nothing but the unique words aka vocabulary** of the numerical representation — ready to be converted into numerical vectors.\n",
    "\n",
    "Before directly jumping into word embedding let’s have a look at some of the **traditional methods of numerical encoding and why they are not good methods for feeding into neural networks.**\n",
    "\n",
    "**Label Encoding**\n",
    "\n",
    "   *Assigns each token in the corpus a unique number.* It is directly proportional to the size of the vocabulary and due to its nature of sequencing a relation is established between the features getting higher numbers and a biased model is inevitable. This could lead to poor performing models and unexpected results, so it’s a big NO for using this technique in terms of feature representation instead is always used for target variables/labels hence the name.\n",
    "    \n",
    "**One Hot Encoder**\n",
    "\n",
    "   *Each token in the corpus vocabulary is given a unique integer representation that is between 1 and n, where n is the length/number of unique tokens in the corpus vocabulary(set of tokens)*. In n-dimensional space, each token would occupy one of the dimensions, meaning none of these tokens has any similarity between them irrespective of their context. This is done via an n-dimension vector filled with all 0s barring the index which is marked as 1.\n",
    "\n",
    "\n",
    "**Dummies**\n",
    "  \n",
    "   *A more flexible way to convert categorical values into dummy/indicator values*. By default get_dummies() provides a one-hot version, in order to get the dummy version we need to pass drop_first=True. We see there is some redundancy in one-hot encoding the n-dimensional vector can be represented in n-1 dimensional long. In the above example if any two cities are 0 then it is obvious the third one is 1 hence the above representation can be further converted to like this.\n",
    "    \n",
    "**Why these are not a good fit for neural networks?**\n",
    "\n",
    "    1.They’re discrete representations — the ability to capture relationships between words is lost.\n",
    "    2.The feature vectors are sparse(with most values being zero for any vector) and high-dimensional(proportional to the size of the vocabulary) representations.\n",
    "    3.They cannot handle OOV(out of vocabulary) words.\n",
    "\n",
    "All of these hampers learning capability and high-dimensionality representation making them computationally inefficient. The Dense representation comes to the rescue.    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "054e824c",
   "metadata": {},
   "source": [
    "### Word Embeddings (Extra)\n",
    "\n",
    "   *An embedding is a representation of the Natural Language that can be learned where words that have the same meaning have a similar representation.* \n",
    "    \n",
    "   For the set of words in a corpus, embedding is a mapping between vector space coming from distributional representation to vector space coming from distributed representation. If we’re given the word “USA,” distributionally similar words could be other countries (e.g., Canada, Germany, India, etc.) or cities in the USA.\n",
    "    \n",
    "   The distributed representation is learned based on the usage of words. This allows words that are used in similar ways to result in having similar representations, naturally capturing their meaning.\n",
    "    \n",
    "   Each word is represented by a real-valued vector, often tens or hundreds of dimensions. This is contrasted to the thousands or millions of dimensions required for sparse word representations, such as a one-hot encoding.\n",
    "    \n",
    "   **Create your own embedding layer**\n",
    "   \n",
    "   In order to create your own word embedding layer, you need to perform the following operations:\n",
    "\n",
    "        1.Text cleaning\n",
    "        2.Tokenization\n",
    "        3.Embedding\n",
    "        \n",
    "   **Step 1: Text Cleaning**\n",
    "\n",
    "   This is the building block and very crucial step, the quality of embedding is proportional to the text cleaning as well. The lesser the noise, the better the model. We can leverage the NLP libraries depending upon the type of cleaning required. In this tutorial, I am not going to cover this step for obvious reasons.\n",
    "\n",
    "   **Step 2: Tokenization**\n",
    "\n",
    "   Tokenization means the direct mapping of tokens to numbers. We will use **TensorFlow’s TextVectorization** for this which is a preprocessing layer that directly maps the text features to integer sequences. It performs the following operations:\n",
    "\n",
    "    1. Standardize each example (usually lowercasing + punctuation stripping)\n",
    "    2. Split each example into substrings (usually words)\n",
    "    3. Recombine substrings into tokens (usually ngrams)\n",
    "    4. Index tokens (associate a unique int value with each token)\n",
    "    5. Transform each example using this index, either into a vector of ints or a dense float vector.\n",
    "\n",
    "   Here are some of the key parameters that need to be taken care off:\n",
    "\n",
    "    max_tokens = how many words are there in the vocabulary.\n",
    "    standardize = what kind of standardization you want to apply like “lower”, “strip_punctuation” or “lower_and_strip_punctuation”.\n",
    "    split = whether to split on whitespace or some specified characters.\n",
    "    ngrams = create groups of n_words, by default None(no groups).\n",
    "    output_sequence_length = how long each sequence should be? \n",
    "   A good practice to define the output_sequence_length is to put a value ≥ the average number of tokens in each sentence\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "fb41ebec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus = ['I love TensorFlow', 'TensorFlow loves Tensors', 'Tensors loves Vectors']\n",
    "round(sum([len(i.split()) for i in corpus]) /len(corpus))\n",
    "\n",
    "#output : 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "7f9e8356",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorflow in c:\\users\\chetan\\anaconda3\\lib\\site-packages (2.14.0)\n",
      "Requirement already satisfied: tensorflow-intel==2.14.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow) (2.14.0)\n",
      "Requirement already satisfied: absl-py>=1.0.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (2.0.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (1.6.3)\n",
      "Requirement already satisfied: flatbuffers>=23.5.26 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (23.5.26)\n",
      "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (0.5.4)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (0.2.0)\n",
      "Requirement already satisfied: h5py>=2.9.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (3.7.0)\n",
      "Requirement already satisfied: libclang>=13.0.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (16.0.6)\n",
      "Requirement already satisfied: ml-dtypes==0.2.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (0.2.0)\n",
      "Requirement already satisfied: numpy>=1.23.5 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (1.23.5)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (3.3.0)\n",
      "Requirement already satisfied: packaging in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (23.0)\n",
      "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (4.24.4)\n",
      "Requirement already satisfied: setuptools in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (67.8.0)\n",
      "Requirement already satisfied: six>=1.12.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (1.16.0)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (2.3.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (4.6.3)\n",
      "Requirement already satisfied: wrapt<1.15,>=1.11.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (1.14.1)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (0.31.0)\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (1.59.0)\n",
      "Requirement already satisfied: tensorboard<2.15,>=2.14 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (2.14.1)\n",
      "Requirement already satisfied: tensorflow-estimator<2.15,>=2.14.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (2.14.0)\n",
      "Requirement already satisfied: keras<2.15,>=2.14.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorflow-intel==2.14.0->tensorflow) (2.14.0)\n",
      "Requirement already satisfied: wheel<1.0,>=0.23.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from astunparse>=1.6.0->tensorflow-intel==2.14.0->tensorflow) (0.38.4)\n",
      "Requirement already satisfied: google-auth<3,>=1.6.3 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (2.23.3)\n",
      "Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (1.0.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (3.4.1)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (2.29.0)\n",
      "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (0.7.2)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (2.2.3)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (5.3.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (0.2.8)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (4.9)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (1.3.1)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (1.26.16)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from requests<3,>=2.21.0->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (2023.5.7)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from werkzeug>=1.0.1->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (2.1.1)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (0.4.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in c:\\users\\chetan\\anaconda3\\lib\\site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard<2.15,>=2.14->tensorflow-intel==2.14.0->tensorflow) (3.2.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "585175f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import TextVectorization\n",
    "vectorizer = TextVectorization(max_tokens = 10,\n",
    "                              standardize = 'lower_and_strip_punctuation',\n",
    "                              split = 'whitespace',\n",
    "                              ngrams = None,\n",
    "                               output_mode= 'int',\n",
    "                               output_sequence_length = 4,\n",
    "                               pad_to_max_tokens = True\n",
    "                              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "7aa7a8e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=int64, numpy=\n",
       "array([[7, 6, 3, 0],\n",
       "       [3, 4, 2, 0],\n",
       "       [2, 4, 5, 0]], dtype=int64)>"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Now it’s time for our text_vectorization instance to adapt the sequences in our corpus.\n",
    "\n",
    "#adapt\n",
    "vectorizer.adapt(corpus)\n",
    "\n",
    "#visualize on whole corpus\n",
    "vectorizer(corpus)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fedd5032",
   "metadata": {},
   "source": [
    "The shape is very important here — (3,4), it represents the number of rows then how long is each sequence. Since we defined our “output_sequence_length = 4” hence despite the sequence length being equal to 3 it is padded to 4 with zeros the same applies for the antipodal i.e even if the sequence is longer than the said size it gets truncated up to the defined limit hence selecting this parameter is tricky. Even if a single token will be passed to this method it proceeds with the same padding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11bdcf1f",
   "metadata": {},
   "source": [
    "**Step 3: Embedding**\n",
    "\n",
    "Finally, here we are at the final step in ***creating word embedding layers***! For this, we will use **TensorFlow’s Embedding**. Again here are some of the key parameters that need to be taken care off:\n",
    "\n",
    "input_dim = Size of the vocabulary.\n",
    "\n",
    "After the vectorizer is fitted to the corpus we can get the length of the vocabulary by using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "8ed91212",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_len = len(vectorizer.get_vocabulary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "c7ea599b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_len"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b31564f6",
   "metadata": {},
   "source": [
    "output_dim = Size of the output embedding vector, a value of 10 will mean each token gets represented by 10 vectors\n",
    "\n",
    "Note: Always try to define vectors that are easily divisible by 8 — speeds up computing\n",
    "\n",
    "input_length = length of the sequences being passed to the embedding layer so it should be equal to the output_sequence_length\n",
    "\n",
    "Now, let’s create the instance of Embedding and fit it on the output of the previous step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "b327ccf8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "embed = Embedding(input_dim = vocab_len,\n",
    "                 output_dim = 4,\n",
    "                 input_length = 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "c5febbc9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4, 4), dtype=float32, numpy=\n",
       "array([[[ 0.0162386 , -0.04755688, -0.01951839, -0.0486214 ],\n",
       "        [ 0.0140211 , -0.00024027, -0.04202429,  0.02287618],\n",
       "        [ 0.02416504,  0.01927711, -0.01489583, -0.03634726],\n",
       "        [ 0.01321355, -0.02965011,  0.00294185,  0.04639276]],\n",
       "\n",
       "       [[ 0.02416504,  0.01927711, -0.01489583, -0.03634726],\n",
       "        [ 0.03322167,  0.02805246,  0.04029535,  0.03265033],\n",
       "        [-0.00534086, -0.0002786 , -0.02870411,  0.02548721],\n",
       "        [ 0.01321355, -0.02965011,  0.00294185,  0.04639276]],\n",
       "\n",
       "       [[-0.00534086, -0.0002786 , -0.02870411,  0.02548721],\n",
       "        [ 0.03322167,  0.02805246,  0.04029535,  0.03265033],\n",
       "        [ 0.03240671,  0.01602124, -0.03174583,  0.00118878],\n",
       "        [ 0.01321355, -0.02965011,  0.00294185,  0.04639276]]],\n",
       "      dtype=float32)>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit on tokenized corpus\n",
    "embed(vectorizer(corpus))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78a34d87",
   "metadata": {},
   "source": [
    "The output shape is very important since this shape will be responsible for feeding this embedding into any neural network.\n",
    "\n",
    "So (3,4,4) what does it mean?\n",
    "\n",
    "3 — three sequences are passed,\n",
    "\n",
    "4 — each sentence is padded to 4 tokens, so all zeros will have the same representation.\n",
    "\n",
    "4 — Each token is represented by 4 feature vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "4606f227",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 4, 4), dtype=float32, numpy=\n",
       "array([[[ 0.02416504,  0.01927711, -0.01489583, -0.03634726],\n",
       "        [ 0.01321355, -0.02965011,  0.00294185,  0.04639276],\n",
       "        [ 0.01321355, -0.02965011,  0.00294185,  0.04639276],\n",
       "        [ 0.01321355, -0.02965011,  0.00294185,  0.04639276]]],\n",
       "      dtype=float32)>"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embed(vectorizer(['TensorFlow']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17054a68",
   "metadata": {},
   "source": [
    "As you can see, the shape this time is (1,4,4). Though a single token is passed it is considered as a sequence and gets padded. Therefore the first row is unique and the rest all have the same representation of 0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "458ee594",
   "metadata": {},
   "source": [
    "Now we know how to create a basic word embeddings layer efficiently in TensorFlow! The two famous algorithms which are just an extended version of this layer are word2vec and GloVe\n",
    "\n",
    "\n",
    "**Summary:**\n",
    "\n",
    "    Neural Networks work with numbers so you can’t just throw words in it.\n",
    "    You could one-hot encode all the words but that will result in a sparse matrix, you will lose the notion of similarity between them — a big NO-NO.\n",
    "    Neural networks work best with dense layers.\n",
    "    Therefore we feed the Embedding layer to the neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "517ccfbf",
   "metadata": {},
   "source": [
    "### Bag of Words & TF-IDF (Text Representation)\n",
    "\n",
    "Text representations techniques:\n",
    "\n",
    "**Bag of Words**\n",
    "\n",
    "This method represents the considered text as a collection of words, disregarding their order and context. It assumes that text belonging to a specific class in the dataset can be characterized by a unique set of words. Essentially, it creates a list of vector arrays for each sentence or corpus, where each word is encoded using a one-hot encoding scheme. The size of the list is determined by the number of unique words in the set, forming a Bag 💼 representation.\n",
    "\n",
    "If two text pieces share similar words, they are considered to belong to the same class or Bag. By analyzing the words present in a text, one can identify the corresponding class it falls into. Consequently, in this representation, documents with the same words will have their vector representations located close to each other in the vector space.\n",
    "\n",
    "However, this approach results in a sparse representation, with the majority of vector entries being zeroes. This sparsity can lead to computational inefficiencies in terms of storage, computation, and learning (overfitting). To overcome these limitations, the next method introduced is TF-IDF.\n",
    "\n",
    "\n",
    "**TF-IDF**\n",
    "\n",
    "In the previous text representation techniques all words within a text are considered equally significant, with no distinction of importance. However, to overcome this limitation, the concept of **TF-IDF (term frequency–inverse document frequency)** emerges. TF-IDF strives to **measure the relative importance of a word by comparing its frequency in a specific document to its occurrence across the entire corpus.** This approach serves as a widely employed representation scheme in information-retrieval systems, aiding in the extraction of relevant documents from a corpus based on a given text query.\n",
    "\n",
    "**TF (term frequency)** measures how often a term or word occurs in a given document and TF of a term t in a document d is given as\n",
    "\n",
    "    TF(t,d) = (Number of occurrences of term t in document d) / (Total number of terms in document d)\n",
    "   \n",
    "   \n",
    "**IDF (inverse document frequency)** measures the importance of the term across a corpus. In computing TF (t), all terms are given equal importance (weightage). **IDF weighs down the terms that are very common across a corpus and weighs up the rare terms**. IDF of a term t is given as\n",
    "\n",
    "    IDF(t) = log[ (Total number of documents in the corpus) / (Number of documents with term t in them) ]\n",
    "    \n",
    "    \n",
    "    TF-IDF = TF(t,d) * IDF(t)\n",
    "\n",
    "The TF-IDF score is calculated for the vocabulary of the corpus(consider the union of unique words across all documents). Then each document is represented as a TF-IDF score of words present to them.\n",
    "\n",
    "Note: Word preprocessing like stop words removal, stemming , lemmatization is crucial here for the obvious reasons (Ex: Eat, Eating, Eats will all be converted into Eat) hence will lower the dimensions of the array."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e75ad7",
   "metadata": {},
   "source": [
    "### Cosine Similarity (Text Similarity)\n",
    "\n",
    "There are multiple distance measures in data science:\n",
    "\n",
    "    1. Euclidean\n",
    "    2. Cosine\n",
    "    3. Hamming\n",
    "    4. Manhattan\n",
    "    5. Minkowski\n",
    "    6. Chebyshev\n",
    "    7. Jaccard\n",
    "    8. Haversine\n",
    "    9. Sorensen-Dice\n",
    "    \n",
    "\n",
    "**Cosine Similairy**:\n",
    "\n",
    "Cosine similarity is a mathematical approach **used to measure the similarity between pairs of vectors or rows of numbers treated as vectors.** It involves representing each value in a sample endpoint coordinates for a vector, with the other endpoint at the origin of the coordinate system. This process is repeated for two samples, and the cosine between the vectors is computed in an m-dimensional space, where m represents the number of values in each sample. **The cosine similarity is such that identical vectors have a similarity of 1 since the cosine of 0 is 1. As the vectors become more dissimilar, the cosine value approaches 0, indicating a lower similarity.**\n",
    "\n",
    "Cosine similarity between two vectors, A and B, each with n components, between them is computed as follows:\n",
    "\n",
    "    similarity = cos(theta) = A.B / ||A||^2 ||B||^2\n",
    "    \n",
    "The dot product of vectors A (x1​, x2​, x3​) and B (y1​, y2​, y3​), denoted as A.B, can be computed using the following formula:\n",
    "\n",
    "A.B = x1​∗y1​ + x2​∗y2 ​+ x3​∗y3 ​\n",
    "\n",
    "and ||A|| & ||B|| is calculated by:\n",
    "\n",
    "∣∣A∣∣ = ✓(​x1²​+x2​²+x3​²)\n",
    "\n",
    "∣∣B∣∣ = ✓(​y1²​+y2​²+y3​²)\n",
    "\n",
    "\n",
    "Let’s consider a simple example in two-dimensional space. It’s worth noting the beauty of mathematics lies in the fact that if we can comprehend a concept in two dimensions, we can extend that understanding to any number of dimensions. Let's walk through the example⚡"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "f531db6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define three vectors\n",
    "A = [1, 2]\n",
    "B = [2, 3]\n",
    "C = [3, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "2b80eecc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate ot products\n",
    "ab = np.dot(A,B)\n",
    "bc = np.dot(B,C)\n",
    "ca = np.dot(C,A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "4943cf11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the length of the vector\n",
    "a = np.linalg.norm(A)\n",
    "b = np.linalg.norm(B)\n",
    "c = np.linalg.norm(C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "26f170f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculte cosine similarity for each pair using the above formula\n",
    "sim_ab = ab/(a*b)\n",
    "sim_bc = bc/(b*c)\n",
    "sim_ca = ca/(c*a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "5faad311",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.9922778767136677, 0.7893522173763263, 0.7071067811865475)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# lets see the similarities\n",
    "sim_ab, sim_bc, sim_ca"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c77ce358",
   "metadata": {},
   "source": [
    "By utilizing the **pairwise.cosine_similarity** class from sklearn, we can effortlessly replicate all these computations with just a single line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a550bf01",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "6d48478d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.        , 0.99227788, 0.70710678],\n",
       "       [0.99227788, 1.        , 0.78935222],\n",
       "       [0.70710678, 0.78935222, 1.        ]])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "# compute cosine similarity of the example above\n",
    "cosine_similarity([A,B,C])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59d20ce3",
   "metadata": {},
   "source": [
    "The cosine_similarity function returns a matrix of cosine similarities, similar to a correlation matrix. Each row and column in the matrix represents a vector, resulting in a diagonal matrix where the diagonal values are always one (since the vectors are being compared to themselves).\n",
    "\n",
    "From the obtained results, it becomes apparent that vector A is more similar to vector B compared to vector C. Similarly, vector B is closer to vector C. To gain a visual understanding, let’s plot these vectors in a 2D space and create a DataFrame using the cosine similarity matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "4bd74caa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.992278</td>\n",
       "      <td>0.707107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>B</th>\n",
       "      <td>0.992278</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.789352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C</th>\n",
       "      <td>0.707107</td>\n",
       "      <td>0.789352</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          A         B         C\n",
       "A  1.000000  0.992278  0.707107\n",
       "B  0.992278  1.000000  0.789352\n",
       "C  0.707107  0.789352  1.000000"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = [A, B, C]\n",
    "pd.DataFrame(cosine_similarity(data), \n",
    "             columns=['A', 'B', 'C'], \n",
    "             index=['A', 'B', 'C'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94c8fbca",
   "metadata": {},
   "source": [
    "## Generating Recommendation Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "333393a3",
   "metadata": {},
   "source": [
    "We will take the DataFrame we have generated and apply TF-IDF vectorization to the corpus column. This process will yield a feature matrix where the rows represent the movies, and the columns correspond to the textual representations of the vocabulary. We can verify the same via shapes of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "7f54feb7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>corpus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>6.1</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                     title  vote_average  \\\n",
       "0   19995                                    Avatar           7.2   \n",
       "1     285  Pirates of the Caribbean: At World's End           6.9   \n",
       "2  206647                                   Spectre           6.3   \n",
       "3   49026                     The Dark Knight Rises           7.6   \n",
       "4   49529                               John Carter           6.1   \n",
       "\n",
       "                                              corpus  \n",
       "0  In the 22nd century, a paraplegic Marine is di...  \n",
       "1  Captain Barbossa, long believed to be dead, ha...  \n",
       "2  A cryptic message from Bond’s past sends him o...  \n",
       "3  Following the death of District Attorney Harve...  \n",
       "4  John Carter is a war-weary, former military ca...  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "75e56b82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In the 22nd century, a paraplegic Marine is dispatched to the moon Pandora on a unique mission, but becomes torn between following orders and protecting an alien civilization. Action Adventure Fantasy Science Fiction Sam Worthington Zoe Saldana Sigourney Weaver Stephen Lang Jon Landau James Cameron'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.corpus[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "9b8fb059",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Initialize the object and remove stopwords\n",
    "tfidf = TfidfVectorizer(stop_words='english')\n",
    "tfidf_matrix = tfidf.fit_transform(df['corpus'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "973d7628",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<4800x30258 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 202256 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f61cc04e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4800, 4)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compare shapes\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d90a77ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4800, 30258)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05e46ad9",
   "metadata": {},
   "source": [
    "Now, we have a situation similar to what we previously discussed with examples A, B, and C, in our current context, we have a dataset comprising 4,800 movies, and the text representation of each movie is described by 30258-word vectors, representing the vocabulary of the entire corpus. To compute the cosine similarity, we need to generate a matrix where the rows and columns represent the 4,800 movies. Each movie will be compared to every other movie along both axes, and their dot product divided by the length of the vectors will yield the cosine similarity values.\n",
    "\n",
    "Given our prior understanding of the basic example, we can envision the matrix we are aiming to generate. However, to avoid manual calculation, we can leverage the **linear kernel provided by scikit-learn**. This handy tool simplifies the process of generating the cosine similarity matrix, saving us considerable effort and time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "c16a51da",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "\n",
    "# compute the similarity matrix\n",
    "cos_mat = linear_kernel(tfidf_matrix, tfidf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "dc6039a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4800, 4800)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cos_mat.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a62c290",
   "metadata": {},
   "source": [
    "We have our similarity matrix. As per our theory, the diagonal elements of the matrix should be 1 since each movie is being compared to itself. To verify this, if we sum up all the diagonal elements of the matrix, it should yield a value of 4800. Let’s put it to the test and confirm our expectations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a9c87c02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4800.0\n"
     ]
    }
   ],
   "source": [
    "diag = 0\n",
    "for i in range(len(cos_mat)):\n",
    "    diag += cos_mat[i][i]\n",
    "    \n",
    "print(diag)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f65dd37",
   "metadata": {},
   "source": [
    "Now that we have everything prepared, here’s how the workflow will unfold: \n",
    "\n",
    "    When the user provides a movie name, the model will locate the corresponding index of the movie in our DataFrame. \n",
    "    We can then use this index to retrieve the same index from the similarity matrix. \n",
    "    As the DataFrame and the cosine matrix are aligned, this step yields an array containing the cosine similarity scores of that movie with all other movies in the database.\n",
    "\n",
    "However, *the array is not sorted in any particular order, and we want to showcase the most similar movies*. To achieve this, we *need to sort the array in descending order*. The first element will always correspond to the movie itself, with a similarity score of 1, followed by the other movies in descending order of similarity. Here lies a challenge: *Sorting the array will disrupt the original order, making it difficult to fetch the movie titles from the database.*\n",
    "\n",
    "To overcome this, *we can store the movie index and similarity score as tuples*. Then, we can perform the sorting based on the score alone while keeping the index intact. Subsequently, we can retrieve the movie details using the index from the tuple, ensuring we maintain the correct movie-title association. This approach allows us to obtain the desired similarity rankings while preserving the necessary information for fetching movie details. We can also pass a parameter n for slicing i.e. to fetch top n similar movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "211ba2e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.9999999999999997),\n",
       " (4399, 0.1415899568738931),\n",
       " (942, 0.13498538684879574),\n",
       " (3603, 0.12258512012128736),\n",
       " (2403, 0.11852902613288109),\n",
       " (838, 0.11581508223430054),\n",
       " (1245, 0.11431512309211009),\n",
       " (740, 0.113079183458059),\n",
       " (43, 0.11196032334004538),\n",
       " (94, 0.11032651423288937),\n",
       " (2138, 0.09253531408518909),\n",
       " (3104, 0.09224403598052139),\n",
       " (634, 0.09154705701674891),\n",
       " (47, 0.09115720100066335),\n",
       " (1341, 0.0902650431649201),\n",
       " (1178, 0.08732362348424455),\n",
       " (529, 0.08591994756833259),\n",
       " (150, 0.08530818125228448),\n",
       " (2966, 0.08072035871975969),\n",
       " (2060, 0.07985113908534044),\n",
       " (1053, 0.07905762857235527),\n",
       " (1796, 0.0784787631654588),\n",
       " (775, 0.07819004432036487),\n",
       " (182, 0.07814662454080266),\n",
       " (68, 0.07788068882087319),\n",
       " (3055, 0.07747171918786659),\n",
       " (2244, 0.07681834061302681),\n",
       " (3157, 0.07463520401253124),\n",
       " (311, 0.07248613632315021),\n",
       " (2130, 0.07204524347418373),\n",
       " (1804, 0.0719901836216657),\n",
       " (3648, 0.0714273250138737),\n",
       " (1092, 0.07114847515784227),\n",
       " (2103, 0.07109389288326345),\n",
       " (2500, 0.07082212708531822),\n",
       " (1213, 0.07037602857987968),\n",
       " (1633, 0.06991154848463368),\n",
       " (1372, 0.06949678748988686),\n",
       " (1574, 0.06917397638871298),\n",
       " (3571, 0.06885505245252554),\n",
       " (1892, 0.06841377335292309),\n",
       " (2350, 0.06829287897054526),\n",
       " (1959, 0.06795955121030473),\n",
       " (1610, 0.067082267250835),\n",
       " (1274, 0.06623195403669302),\n",
       " (31, 0.06562128156899133),\n",
       " (847, 0.06558113100017356),\n",
       " (3623, 0.06448343075414714),\n",
       " (365, 0.06443176689264413),\n",
       " (36, 0.06416521643672245),\n",
       " (856, 0.06277799595066262),\n",
       " (2198, 0.06264143923755479),\n",
       " (1309, 0.06256381724698078),\n",
       " (174, 0.06179909492329383),\n",
       " (2777, 0.061405400808401345),\n",
       " (2106, 0.0613430109099186),\n",
       " (2361, 0.0607774547814869),\n",
       " (1172, 0.060573164259776414),\n",
       " (1146, 0.060528888938530784),\n",
       " (2509, 0.06052111564262834),\n",
       " (473, 0.05997689062761665),\n",
       " (972, 0.059405926366220496),\n",
       " (206, 0.05934822694671103),\n",
       " (417, 0.05922379535351192),\n",
       " (2780, 0.05912998245020644),\n",
       " (646, 0.05891075752131566),\n",
       " (4038, 0.05889163155195923),\n",
       " (1652, 0.05880078959800955),\n",
       " (1400, 0.058781887518538634),\n",
       " (1914, 0.0587325866882845),\n",
       " (1016, 0.05797218484634364),\n",
       " (1617, 0.057758976991937334),\n",
       " (3438, 0.057636094853519934),\n",
       " (557, 0.057096687979205536),\n",
       " (5, 0.057051005952008745),\n",
       " (107, 0.05700492133676062),\n",
       " (7, 0.05672427466073207),\n",
       " (131, 0.056499886933023456),\n",
       " (83, 0.05634102507141208),\n",
       " (475, 0.05558789299099473),\n",
       " (1138, 0.05545812435790028),\n",
       " (505, 0.054886606294666844),\n",
       " (539, 0.05476908195296159),\n",
       " (45, 0.054559696107800176),\n",
       " (3947, 0.05454334985455606),\n",
       " (4299, 0.054415333084548885),\n",
       " (3483, 0.05438451026340604),\n",
       " (10, 0.05434568755133462),\n",
       " (855, 0.05420276659844035),\n",
       " (74, 0.054085883795119756),\n",
       " (4041, 0.05403157300685761),\n",
       " (1068, 0.05338369539576272),\n",
       " (2424, 0.053283342649796075),\n",
       " (1500, 0.05322809492720707),\n",
       " (483, 0.05315824728020767),\n",
       " (1482, 0.053072814500218696),\n",
       " (2327, 0.05301285493132008),\n",
       " (1448, 0.05297016612865008),\n",
       " (2368, 0.052553155946053476),\n",
       " (369, 0.05225838344177528),\n",
       " (3335, 0.051845839241361207),\n",
       " (2614, 0.05179047404095875),\n",
       " (2578, 0.05159993760151799),\n",
       " (1495, 0.05110850281936004),\n",
       " (495, 0.05109184419795837),\n",
       " (507, 0.05098683978609771),\n",
       " (39, 0.050840424155613406),\n",
       " (2826, 0.05078141475742759),\n",
       " (466, 0.05072537352820484),\n",
       " (587, 0.05067224282640666),\n",
       " (111, 0.050656009197922604),\n",
       " (1412, 0.05061840716281797),\n",
       " (3714, 0.05057890757752059),\n",
       " (27, 0.05035858047590941),\n",
       " (559, 0.05020363829062171),\n",
       " (312, 0.050195492882912945),\n",
       " (2584, 0.04994922080754339),\n",
       " (1802, 0.049863920398632924),\n",
       " (3473, 0.04918639728649346),\n",
       " (582, 0.04915731215511357),\n",
       " (279, 0.048926291698748424),\n",
       " (281, 0.048609384268062494),\n",
       " (1034, 0.04857334642266772),\n",
       " (1275, 0.04844540004030527),\n",
       " (25, 0.04840222572845274),\n",
       " (575, 0.04821826291493154),\n",
       " (1317, 0.04817617468309925),\n",
       " (199, 0.04809304877897128),\n",
       " (149, 0.04800313615393237),\n",
       " (3617, 0.04797126489357177),\n",
       " (1644, 0.04792524396924962),\n",
       " (46, 0.04788055055877512),\n",
       " (2027, 0.047836304391801386),\n",
       " (4185, 0.04780910448878365),\n",
       " (1198, 0.047425928777971665),\n",
       " (1259, 0.04727595016215966),\n",
       " (1438, 0.04721412664412896),\n",
       " (1735, 0.04686924023739198),\n",
       " (4174, 0.0467498708548648),\n",
       " (2153, 0.04652331054480787),\n",
       " (151, 0.04645856217923393),\n",
       " (127, 0.046400222033084845),\n",
       " (3606, 0.046333643965054835),\n",
       " (420, 0.04627661306908319),\n",
       " (1900, 0.04620862102454157),\n",
       " (570, 0.04617717627744364),\n",
       " (2638, 0.046125573837502695),\n",
       " (26, 0.04593083590577745),\n",
       " (3642, 0.045823055653515604),\n",
       " (1191, 0.045619804345814766),\n",
       " (1367, 0.045463332999538295),\n",
       " (932, 0.04537863972682459),\n",
       " (2114, 0.04526231087577684),\n",
       " (2963, 0.045242078837719864),\n",
       " (1784, 0.045241444920299925),\n",
       " (70, 0.0452401011728055),\n",
       " (2636, 0.045231947643472756),\n",
       " (1318, 0.04520412442149987),\n",
       " (461, 0.045173720642733256),\n",
       " (403, 0.04516838792540307),\n",
       " (200, 0.0451102670694181),\n",
       " (91, 0.04509290001946613),\n",
       " (2391, 0.04502757392836729),\n",
       " (985, 0.04492554406176898),\n",
       " (373, 0.04485431923613301),\n",
       " (899, 0.044702970074704904),\n",
       " (637, 0.044569529500113364),\n",
       " (140, 0.04434357754079009),\n",
       " (1326, 0.04420278002669545),\n",
       " (2218, 0.04405092389396373),\n",
       " (1201, 0.04396871171659337),\n",
       " (232, 0.043918363818821735),\n",
       " (1033, 0.04384016209930594),\n",
       " (132, 0.04361315735500667),\n",
       " (4758, 0.04356326783658723),\n",
       " (3723, 0.043554712740922014),\n",
       " (581, 0.04337057855453895),\n",
       " (1218, 0.043297058316686704),\n",
       " (1099, 0.0432777899619997),\n",
       " (3069, 0.043273065834416875),\n",
       " (165, 0.04322683182096723),\n",
       " (894, 0.04320164516990688),\n",
       " (2262, 0.0431478032250682),\n",
       " (1518, 0.043099911290717156),\n",
       " (579, 0.04287807754021499),\n",
       " (1199, 0.04285183710173633),\n",
       " (1158, 0.042744611339659164),\n",
       " (2543, 0.04271119307003836),\n",
       " (159, 0.04265871566567571),\n",
       " (2874, 0.04242504815538743),\n",
       " (35, 0.04236426192515854),\n",
       " (1737, 0.042338507912992755),\n",
       " (1013, 0.0422881002583342),\n",
       " (823, 0.04217871615904943),\n",
       " (1491, 0.042054364841759326),\n",
       " (487, 0.04204191881327378),\n",
       " (422, 0.04202523142303303),\n",
       " (166, 0.04199156799496727),\n",
       " (227, 0.04190383618832648),\n",
       " (626, 0.04184998466427101),\n",
       " (2529, 0.04183305466881285),\n",
       " (322, 0.041711350713760566),\n",
       " (2312, 0.04146620775024457),\n",
       " (568, 0.04146316673370207),\n",
       " (2573, 0.04144728480420487),\n",
       " (2796, 0.04113972765963637),\n",
       " (412, 0.04113692513234846),\n",
       " (1531, 0.04107036562927855),\n",
       " (1715, 0.040939517806766815),\n",
       " (1654, 0.04083671183380489),\n",
       " (2017, 0.04080130753204399),\n",
       " (3086, 0.04079898240951983),\n",
       " (3969, 0.04075512530689973),\n",
       " (102, 0.04064570032880857),\n",
       " (282, 0.04063405967093971),\n",
       " (79, 0.04063297414956471),\n",
       " (931, 0.04055706882601969),\n",
       " (1067, 0.04046794456562785),\n",
       " (778, 0.04044582779874867),\n",
       " (2220, 0.04033413476688786),\n",
       " (2814, 0.0403100270732453),\n",
       " (1913, 0.04029585350814766),\n",
       " (37, 0.040266244449978776),\n",
       " (1671, 0.040223313920268365),\n",
       " (1265, 0.040139710189466306),\n",
       " (3396, 0.04003553808388628),\n",
       " (2482, 0.039979830206198136),\n",
       " (1093, 0.039851114615228486),\n",
       " (1494, 0.039847855243180394),\n",
       " (2920, 0.03983038999076179),\n",
       " (920, 0.03964926312392184),\n",
       " (1271, 0.0393615564260562),\n",
       " (467, 0.0392830848700268),\n",
       " (2725, 0.03916917693444363),\n",
       " (2226, 0.03913567065157567),\n",
       " (1402, 0.039119954194911724),\n",
       " (3622, 0.0390471577751948),\n",
       " (4793, 0.03880005237364764),\n",
       " (2011, 0.03879858932214697),\n",
       " (395, 0.03864478627345557),\n",
       " (1513, 0.03863288825249298),\n",
       " (3694, 0.038486071671455784),\n",
       " (1653, 0.03826224848803784),\n",
       " (661, 0.03819265939530616),\n",
       " (3533, 0.03813544288359588),\n",
       " (2264, 0.03811209850930667),\n",
       " (564, 0.03797486576072798),\n",
       " (4261, 0.037893774033601155),\n",
       " (275, 0.037853431158958784),\n",
       " (1651, 0.03739148267994165),\n",
       " (30, 0.03738798677759429),\n",
       " (3667, 0.0373459820691014),\n",
       " (502, 0.03726303601367918),\n",
       " (2314, 0.03714185414250325),\n",
       " (2864, 0.03709159931510468),\n",
       " (2989, 0.03705531472239491),\n",
       " (4209, 0.03698207417081403),\n",
       " (76, 0.03692060216939831),\n",
       " (513, 0.036859237713144706),\n",
       " (1345, 0.036821814769263626),\n",
       " (3655, 0.03680502937698718),\n",
       " (2163, 0.036749752807705274),\n",
       " (715, 0.03672441323907755),\n",
       " (1415, 0.03672415130325307),\n",
       " (14, 0.03671259739035685),\n",
       " (3423, 0.036669023860319915),\n",
       " (29, 0.03664471734752177),\n",
       " (2766, 0.03650731645112356),\n",
       " (1583, 0.03645931202716629),\n",
       " (3504, 0.03644793082566238),\n",
       " (549, 0.03640932544807862),\n",
       " (1293, 0.03640357073582855),\n",
       " (2382, 0.03638941215057845),\n",
       " (1999, 0.03637444416581813),\n",
       " (3992, 0.03632356964767056),\n",
       " (342, 0.03630475126008661),\n",
       " (4246, 0.036283731998612255),\n",
       " (3135, 0.03624472872580514),\n",
       " (3150, 0.03622929593332385),\n",
       " (148, 0.0362064783781782),\n",
       " (2157, 0.036197273425502455),\n",
       " (2645, 0.03605595091702491),\n",
       " (4048, 0.035963459341487),\n",
       " (1736, 0.03585345154217563),\n",
       " (3395, 0.03585331761559242),\n",
       " (3746, 0.03582614791387017),\n",
       " (3003, 0.0357959778085138),\n",
       " (1272, 0.035772297473798786),\n",
       " (4508, 0.03573225440146832),\n",
       " (1825, 0.03571825286417858),\n",
       " (334, 0.035698523069753664),\n",
       " (3492, 0.03568986289228538),\n",
       " (929, 0.03561633152085215),\n",
       " (1667, 0.03549394672813533),\n",
       " (23, 0.0354414160475398),\n",
       " (1377, 0.03535152358940789),\n",
       " (4496, 0.03526694932705475),\n",
       " (1692, 0.035067432392282896),\n",
       " (315, 0.03506668872637154),\n",
       " (52, 0.0350337882734963),\n",
       " (236, 0.035032577633083886),\n",
       " (3567, 0.03499706962035326),\n",
       " (4783, 0.03478782731300006),\n",
       " (2069, 0.034764089548344375),\n",
       " (3098, 0.03472055636042937),\n",
       " (61, 0.03456614443277087),\n",
       " (2006, 0.03452889948525433),\n",
       " (951, 0.03450266830057114),\n",
       " (3457, 0.034475779536385955),\n",
       " (1217, 0.03435679183932172),\n",
       " (2317, 0.034305176877908385),\n",
       " (1301, 0.03423306984828738),\n",
       " (2616, 0.03420805952142334),\n",
       " (4353, 0.034202128489464786),\n",
       " (4614, 0.034190232887849094),\n",
       " (184, 0.03406986115590141),\n",
       " (1750, 0.03400500992302063),\n",
       " (1786, 0.03394425996938518),\n",
       " (103, 0.03392222261691071),\n",
       " (1277, 0.03389245760202815),\n",
       " (2071, 0.033839881962458365),\n",
       " (101, 0.033767466654660024),\n",
       " (1528, 0.033750458210338286),\n",
       " (632, 0.033747146971291705),\n",
       " (3342, 0.03372858157093095),\n",
       " (1385, 0.03359188407934183),\n",
       " (3350, 0.03352042727709941),\n",
       " (4539, 0.03348705944491216),\n",
       " (329, 0.033479231033914245),\n",
       " (2038, 0.03347715543132122),\n",
       " (3305, 0.03346696752848746),\n",
       " (4158, 0.03343599143625176),\n",
       " (1390, 0.03339333944900101),\n",
       " (87, 0.033336846919922176),\n",
       " (228, 0.033323807712400524),\n",
       " (4780, 0.03325829183199391),\n",
       " (4572, 0.033180571827112704),\n",
       " (813, 0.03317802407432579),\n",
       " (3359, 0.0331701241631565),\n",
       " (3385, 0.033023091776364484),\n",
       " (597, 0.03299990410748244),\n",
       " (169, 0.032983448239887074),\n",
       " (1843, 0.032973160046164185),\n",
       " (2550, 0.03296125083538949),\n",
       " (4667, 0.03286736620512133),\n",
       " (2066, 0.03285822197298892),\n",
       " (300, 0.03276895230641245),\n",
       " (72, 0.032755565111294925),\n",
       " (270, 0.032694926089046956),\n",
       " (2193, 0.03268531065910081),\n",
       " (2551, 0.032654446416072604),\n",
       " (1392, 0.03261053713946752),\n",
       " (1939, 0.032593212463597546),\n",
       " (1434, 0.032584044562825855),\n",
       " (1020, 0.032528574320917956),\n",
       " (1106, 0.03250258000476701),\n",
       " (3493, 0.03245974976255052),\n",
       " (3301, 0.032440828812835824),\n",
       " (3196, 0.03241796031104679),\n",
       " (106, 0.032404830655080125),\n",
       " (1611, 0.032390709365601324),\n",
       " (2628, 0.0323862561682339),\n",
       " (3097, 0.03237965854596156),\n",
       " (1921, 0.032355821617415195),\n",
       " (249, 0.03233169748466697),\n",
       " (2752, 0.03231916681208831),\n",
       " (2491, 0.0322245945460906),\n",
       " (1782, 0.032198970653267066),\n",
       " (1002, 0.032186179448841605),\n",
       " (363, 0.03216720829825841),\n",
       " (454, 0.03207943896445735),\n",
       " (535, 0.03206803607487031),\n",
       " (617, 0.032058641812501926),\n",
       " (86, 0.032047606585828316),\n",
       " (2807, 0.03202534137244122),\n",
       " (1359, 0.03200816668521561),\n",
       " (158, 0.031878516921398314),\n",
       " (3001, 0.03180831688588048),\n",
       " (3123, 0.03180217897504825),\n",
       " (896, 0.031785122610847726),\n",
       " (3641, 0.03168566821430135),\n",
       " (3701, 0.031662397314445104),\n",
       " (2541, 0.03160943799510485),\n",
       " (372, 0.03156932820690572),\n",
       " (4596, 0.03155320440830897),\n",
       " (810, 0.031529385463800186),\n",
       " (1902, 0.031514374861234026),\n",
       " (2258, 0.03145997038330135),\n",
       " (3426, 0.03143902222697964),\n",
       " (1753, 0.03136549556468647),\n",
       " (946, 0.03136528655678797),\n",
       " (1957, 0.03135637518813604),\n",
       " (71, 0.03132756462408318),\n",
       " (2844, 0.03130154222459147),\n",
       " (1044, 0.03129141203204187),\n",
       " (1942, 0.03127999126226194),\n",
       " (3017, 0.03126737751215065),\n",
       " (3625, 0.031238305960763674),\n",
       " (2366, 0.031199145399287996),\n",
       " (80, 0.03117518291324361),\n",
       " (1626, 0.03112503074184686),\n",
       " (264, 0.03110861730586078),\n",
       " (614, 0.03108502773009545),\n",
       " (1111, 0.031006461686111438),\n",
       " (1155, 0.030960636691001416),\n",
       " (1188, 0.0309260174867677),\n",
       " (598, 0.0308672091748152),\n",
       " (1319, 0.03076617714358342),\n",
       " (1279, 0.03076570435319414),\n",
       " (1117, 0.03073169801538759),\n",
       " (4751, 0.030720305954090407),\n",
       " (4118, 0.030719936257857038),\n",
       " (139, 0.030685592898857753),\n",
       " (78, 0.030685320647091147),\n",
       " (219, 0.030682924087977237),\n",
       " (2109, 0.03067739095386621),\n",
       " (1547, 0.030673642374689367),\n",
       " (476, 0.030640027638936064),\n",
       " (2850, 0.030632397914340408),\n",
       " (3418, 0.030630996017133795),\n",
       " (1395, 0.03061792320998544),\n",
       " (984, 0.03061497610026959),\n",
       " (4593, 0.030612704162690543),\n",
       " (330, 0.0305907544711186),\n",
       " (2877, 0.030559134242004547),\n",
       " (870, 0.030530322152957916),\n",
       " (332, 0.030520798690896578),\n",
       " (545, 0.03051977443957568),\n",
       " (1936, 0.030487376229919455),\n",
       " (241, 0.030478429595173512),\n",
       " (4450, 0.030452580167151133),\n",
       " (756, 0.030422569702863823),\n",
       " (3987, 0.030420455166665132),\n",
       " (1569, 0.030402097927854928),\n",
       " (2540, 0.030400982412955323),\n",
       " (358, 0.030386207709153745),\n",
       " (4721, 0.030308414778269274),\n",
       " (233, 0.030275159444263504),\n",
       " (3709, 0.03013256093519277),\n",
       " (2127, 0.030096373130792678),\n",
       " (2683, 0.030070091771903017),\n",
       " (4765, 0.029944116142112648),\n",
       " (4447, 0.029939557244517894),\n",
       " (675, 0.029884543661152528),\n",
       " (400, 0.02986554806998899),\n",
       " (782, 0.02972674818112113),\n",
       " (3296, 0.02962826405111477),\n",
       " (977, 0.02961075715390994),\n",
       " (3936, 0.02960350856505716),\n",
       " (2862, 0.02943910655956717),\n",
       " (2427, 0.029439018082153642),\n",
       " (595, 0.029375548896508275),\n",
       " (1869, 0.029323573063858455),\n",
       " (607, 0.02928734949041012),\n",
       " (1352, 0.0292843664288511),\n",
       " (1230, 0.029255645373226988),\n",
       " (3149, 0.02923025458891627),\n",
       " (1369, 0.029194441672627018),\n",
       " (511, 0.029185145811878754),\n",
       " (1192, 0.02916689781132551),\n",
       " (64, 0.029143329671163024),\n",
       " (404, 0.02908936705377896),\n",
       " (1246, 0.029078241676071023),\n",
       " (1175, 0.029065404426554527),\n",
       " (256, 0.029058379022576724),\n",
       " (2375, 0.029028553310589678),\n",
       " (770, 0.029015372628036124),\n",
       " (183, 0.028944141982619232),\n",
       " (787, 0.02890264373077231),\n",
       " (1118, 0.02889848011551402),\n",
       " (1296, 0.028884215068717894),\n",
       " (4249, 0.02887927217051912),\n",
       " (926, 0.02882527840681149),\n",
       " (854, 0.02881204661941625),\n",
       " (426, 0.028804738701587115),\n",
       " (939, 0.028631241329716096),\n",
       " (11, 0.028623035088507415),\n",
       " (3555, 0.02860530127942335),\n",
       " (618, 0.028459788478405433),\n",
       " (2943, 0.02843330249326715),\n",
       " (2626, 0.028373591003975643),\n",
       " (168, 0.028301941752119468),\n",
       " (1678, 0.028255708572544204),\n",
       " (3803, 0.02823778445988842),\n",
       " (1001, 0.028217668893511617),\n",
       " (2720, 0.028205528582217337),\n",
       " (1294, 0.028198168947056103),\n",
       " (3047, 0.02817145801899639),\n",
       " (1449, 0.028170976957488374),\n",
       " (2707, 0.028075835200121385),\n",
       " (4397, 0.02792493091181082),\n",
       " (1840, 0.027914958774303493),\n",
       " (835, 0.027885754630125877),\n",
       " (33, 0.02788441527691817),\n",
       " (4241, 0.02788056624422662),\n",
       " (3349, 0.02785376593704622),\n",
       " (56, 0.02781418673788004),\n",
       " (123, 0.027813756087161086),\n",
       " (3919, 0.027781961064920487),\n",
       " (584, 0.027686943999805633),\n",
       " (134, 0.0276763556395559),\n",
       " (1594, 0.027629003366607334),\n",
       " (4108, 0.027606745649053403),\n",
       " (2237, 0.027569645442168646),\n",
       " (1595, 0.02755854499329099),\n",
       " (2194, 0.027485471261070527),\n",
       " (122, 0.027400105559429948),\n",
       " (2016, 0.027375879579198566),\n",
       " (3389, 0.027372355059369375),\n",
       " (301, 0.027360682551596512),\n",
       " (1284, 0.027339895082859025),\n",
       " (2775, 0.02728517698059407),\n",
       " (4342, 0.027282826232237777),\n",
       " (1012, 0.0271876793622114),\n",
       " (1252, 0.02710644891635922),\n",
       " (4374, 0.02709004990088791),\n",
       " (3733, 0.027085825367973788),\n",
       " (1836, 0.02708174964441967),\n",
       " (3545, 0.02707677780730568),\n",
       " (1579, 0.02706122229414795),\n",
       " (305, 0.026886522757103657),\n",
       " (3548, 0.026782011311941627),\n",
       " (2076, 0.02678104282985488),\n",
       " (3832, 0.026765367951129715),\n",
       " (3719, 0.026759232344519476),\n",
       " (565, 0.026740594974264204),\n",
       " (2117, 0.026690632119146757),\n",
       " (1817, 0.026650865590310005),\n",
       " (1743, 0.026639288827748048),\n",
       " (3930, 0.02659814700597897),\n",
       " (2747, 0.026579040893879406),\n",
       " (1238, 0.02656970184451332),\n",
       " (816, 0.026560812595348378),\n",
       " (2184, 0.026554218182851388),\n",
       " (2250, 0.026542240397526326),\n",
       " (1918, 0.02650743742202475),\n",
       " (1467, 0.02650032163746395),\n",
       " (1079, 0.026494737134028626),\n",
       " (1163, 0.02647468139676739),\n",
       " (488, 0.026445825282452005),\n",
       " (850, 0.026438529903844203),\n",
       " (1568, 0.026427194053896735),\n",
       " (266, 0.026421547815558437),\n",
       " (3749, 0.026401368774524595),\n",
       " (1110, 0.02640043145179754),\n",
       " (672, 0.026382947396590885),\n",
       " (1649, 0.026260640453236127),\n",
       " (2169, 0.02625989218041238),\n",
       " (3510, 0.02620902256205855),\n",
       " (2008, 0.02620428678845532),\n",
       " (2160, 0.026197353631256186),\n",
       " (3638, 0.026137880186113547),\n",
       " (16, 0.02612369885925901),\n",
       " (596, 0.026120080441771436),\n",
       " (4123, 0.02610481071655637),\n",
       " (370, 0.02603206508278447),\n",
       " (533, 0.025974949736495487),\n",
       " (2994, 0.0259240800378519),\n",
       " (2658, 0.025919380367389805),\n",
       " (3276, 0.025879803171039252),\n",
       " (1997, 0.025852075979346967),\n",
       " (3615, 0.02583979668453338),\n",
       " (1410, 0.02580910436521078),\n",
       " (413, 0.025802640038742376),\n",
       " (712, 0.02575814673655582),\n",
       " (859, 0.02568413693595494),\n",
       " (1506, 0.025661205516457404),\n",
       " (4054, 0.025651626644629943),\n",
       " (577, 0.025624282188996247),\n",
       " (922, 0.025613937947366106),\n",
       " (1932, 0.025583913868025103),\n",
       " (1468, 0.025575706812973227),\n",
       " (654, 0.025566925444047883),\n",
       " (2589, 0.02556396193317396),\n",
       " (350, 0.025558672736426008),\n",
       " (4651, 0.025484609684958037),\n",
       " (3928, 0.025473480521964895),\n",
       " (4406, 0.0254096951820968),\n",
       " (230, 0.02531081403731989),\n",
       " (3978, 0.025301561351988563),\n",
       " (4666, 0.025289278579055184),\n",
       " (4326, 0.02526894170815804),\n",
       " (1976, 0.025256730578102064),\n",
       " (3683, 0.025248930935173046),\n",
       " (902, 0.025220095260318265),\n",
       " (1806, 0.02515674572917425),\n",
       " (2590, 0.025145797947187305),\n",
       " (2547, 0.025077004184227807),\n",
       " (1827, 0.0250768329532149),\n",
       " (1185, 0.025071501656649307),\n",
       " (871, 0.025037907126275356),\n",
       " (960, 0.024999332569105488),\n",
       " (1920, 0.024940879244276806),\n",
       " (1830, 0.02493771994630747),\n",
       " (2, 0.02493681610199215),\n",
       " (419, 0.024859350015376935),\n",
       " (4536, 0.024855377864244542),\n",
       " (1473, 0.02481290180943756),\n",
       " (3102, 0.02480133223068757),\n",
       " (242, 0.024796975933307274),\n",
       " (2811, 0.024779771748242516),\n",
       " (4265, 0.02475819276924665),\n",
       " (728, 0.024721412448094605),\n",
       " (238, 0.024707133138233674),\n",
       " (793, 0.024670863733736264),\n",
       " (4311, 0.02466667857585358),\n",
       " (3972, 0.024618592884237917),\n",
       " (3171, 0.02460214277962928),\n",
       " (2950, 0.024517518892703167),\n",
       " (207, 0.024494772258216654),\n",
       " (3311, 0.024460857003231418),\n",
       " (28, 0.024452811159162823),\n",
       " (2825, 0.02442225257320767),\n",
       " (446, 0.024368251463853897),\n",
       " (3586, 0.024363443320414402),\n",
       " (3356, 0.02434215209290741),\n",
       " (220, 0.024303324056642123),\n",
       " (375, 0.02421845589603435),\n",
       " (1348, 0.024210630698434724),\n",
       " (1239, 0.024189297348047815),\n",
       " (104, 0.024186519271028384),\n",
       " (425, 0.024116625649824874),\n",
       " (1209, 0.02408098932251291),\n",
       " (3363, 0.024079645241963793),\n",
       " (893, 0.024051039348279388),\n",
       " (3515, 0.023979572316809684),\n",
       " (986, 0.023942777793254752),\n",
       " (410, 0.02392832331881225),\n",
       " (1303, 0.02390871062047399),\n",
       " (814, 0.023902066284994538),\n",
       " (3658, 0.02389285248005759),\n",
       " (3836, 0.023882846714136905),\n",
       " (96, 0.023878724682220905),\n",
       " (2919, 0.02387351680123463),\n",
       " (1414, 0.0238693578830597),\n",
       " (278, 0.02385575037153497),\n",
       " (934, 0.023785436466340782),\n",
       " (576, 0.023772128370784552),\n",
       " (3103, 0.023757352104929692),\n",
       " (2442, 0.02373939161339253),\n",
       " (143, 0.023737356345099478),\n",
       " (541, 0.023732951630799293),\n",
       " (335, 0.023732111639971992),\n",
       " (2760, 0.02370748385880777),\n",
       " (1777, 0.023700147349684487),\n",
       " (1042, 0.02369511028925504),\n",
       " (3637, 0.023688091376119023),\n",
       " (2873, 0.02368263247550596),\n",
       " (1863, 0.023666096130461153),\n",
       " (3898, 0.02364640881262954),\n",
       " (2155, 0.023630282996757998),\n",
       " (170, 0.023621108849369372),\n",
       " (222, 0.023618958611903856),\n",
       " (4798, 0.0235619404832886),\n",
       " (3127, 0.02354959168200909),\n",
       " (4003, 0.023544793832863974),\n",
       " (4325, 0.023495181318582088),\n",
       " (2965, 0.02347086650913096),\n",
       " (105, 0.02346273159245029),\n",
       " (2142, 0.023425481977006057),\n",
       " (4044, 0.023359344917210322),\n",
       " (2914, 0.023323466518506104),\n",
       " (999, 0.023291313705166884),\n",
       " (490, 0.02329114548814267),\n",
       " (4262, 0.023267669641409156),\n",
       " (2581, 0.02324688079450419),\n",
       " (2723, 0.023216210490632935),\n",
       " (3220, 0.023213428596719227),\n",
       " (4648, 0.023203729635441664),\n",
       " (1085, 0.02318725505083101),\n",
       " (108, 0.023157930333801133),\n",
       " (3894, 0.02312782299762876),\n",
       " (1725, 0.02312171605359972),\n",
       " (2577, 0.023113352758854733),\n",
       " (3142, 0.023106394449121215),\n",
       " (4643, 0.023078637288246447),\n",
       " (2605, 0.023057774596902184),\n",
       " (1074, 0.023031205772374917),\n",
       " (4021, 0.023018395743767423),\n",
       " (116, 0.02300907260860337),\n",
       " (4228, 0.02298157536117585),\n",
       " (508, 0.022973246291958925),\n",
       " (2875, 0.022941959409570074),\n",
       " (185, 0.022925300586249645),\n",
       " (4011, 0.022922947676466184),\n",
       " (2444, 0.022917118656949165),\n",
       " (1643, 0.022905989253127194),\n",
       " (1056, 0.022894624723918747),\n",
       " (4650, 0.022874967669440503),\n",
       " (2279, 0.022827277761878643),\n",
       " (615, 0.022827019099322905),\n",
       " (51, 0.022812305608373877),\n",
       " (3774, 0.022802123259886013),\n",
       " (1004, 0.022797156507920184),\n",
       " (112, 0.02279600938802049),\n",
       " (2986, 0.02278975485588489),\n",
       " (2320, 0.02275239112383813),\n",
       " (2858, 0.022709408001179415),\n",
       " (768, 0.022707670228961356),\n",
       " (4027, 0.022694977355860485),\n",
       " (1514, 0.022685121621947156),\n",
       " (4731, 0.022678933059435843),\n",
       " (17, 0.02264037229729312),\n",
       " (1889, 0.022605288294386107),\n",
       " (1555, 0.022563738647973286),\n",
       " (4243, 0.022552815936413),\n",
       " (1435, 0.022544517826034432),\n",
       " (1059, 0.022521548284524802),\n",
       " (1822, 0.02251284795919868),\n",
       " (1285, 0.022510491562649317),\n",
       " (3101, 0.022505685217727858),\n",
       " (1658, 0.022432789876173694),\n",
       " (2370, 0.022429172465389054),\n",
       " (3304, 0.022428289704240826),\n",
       " (4478, 0.02242602950105559),\n",
       " (213, 0.022420877847590094),\n",
       " (1302, 0.022415202996294348),\n",
       " (1808, 0.022408999164611285),\n",
       " (785, 0.02236746462813918),\n",
       " (761, 0.022367386147741725),\n",
       " (3840, 0.022358208772342462),\n",
       " (546, 0.022314240160160356),\n",
       " (4, 0.02224263133286682),\n",
       " (3078, 0.022202919803281727),\n",
       " (1083, 0.0221408505600133),\n",
       " (3583, 0.022132003269827578),\n",
       " (2047, 0.022120128977446283),\n",
       " (271, 0.022102942473118174),\n",
       " (4425, 0.022057541222310493),\n",
       " (67, 0.022024922246075156),\n",
       " (187, 0.021996286873377274),\n",
       " (2196, 0.02195901424767421),\n",
       " (3814, 0.021905922882366598),\n",
       " (1463, 0.021878192591122416),\n",
       " (2644, 0.02187543856017572),\n",
       " (4220, 0.02185866829788611),\n",
       " (196, 0.021855461719161494),\n",
       " (239, 0.02184232826693909),\n",
       " (2084, 0.021838237919857048),\n",
       " (1344, 0.02183430995875375),\n",
       " (4784, 0.0217625296255767),\n",
       " (2583, 0.021751022081368306),\n",
       " (2108, 0.02171026952648349),\n",
       " (1365, 0.021703251764844854),\n",
       " (4646, 0.021687250646149035),\n",
       " (2911, 0.021677298456726307),\n",
       " (4073, 0.02167388289021245),\n",
       " (2043, 0.021659172623900994),\n",
       " (723, 0.021645890804786287),\n",
       " (3067, 0.021640689068664204),\n",
       " (2490, 0.021633813233078295),\n",
       " (3303, 0.02161955750795604),\n",
       " (3788, 0.021606767655541117),\n",
       " (470, 0.021593525456530944),\n",
       " (1031, 0.021581893450646823),\n",
       " (1763, 0.021537861760307867),\n",
       " (794, 0.02153358973612408),\n",
       " (84, 0.021500199892809928),\n",
       " (212, 0.021485916795759015),\n",
       " (1086, 0.02146947198152943),\n",
       " (260, 0.021447271450431477),\n",
       " (773, 0.02144617843896069),\n",
       " (1958, 0.021439168289894624),\n",
       " (203, 0.02140184966071969),\n",
       " (291, 0.021376843623487678),\n",
       " (485, 0.021366193268682922),\n",
       " (2188, 0.021327530703821616),\n",
       " (804, 0.021297418383436045),\n",
       " (4562, 0.021278377836683383),\n",
       " (1471, 0.02125667210747054),\n",
       " (734, 0.02124692817180293),\n",
       " (396, 0.021217269843162002),\n",
       " (362, 0.021193689565898503),\n",
       " (976, 0.02117880411866713),\n",
       " (307, 0.021173634951121692),\n",
       " (432, 0.021161660298945024),\n",
       " (4588, 0.021149475705144232),\n",
       " (41, 0.021145365331222278),\n",
       " (3, 0.02114289078935705),\n",
       " (349, 0.021137299865483884),\n",
       " (136, 0.021072616438890857),\n",
       " (12, 0.021051302949615467),\n",
       " (2132, 0.02104031570628453),\n",
       " (4105, 0.02103047555400185),\n",
       " (732, 0.020965026492400337),\n",
       " (2898, 0.020959079478409817),\n",
       " (175, 0.020937790528034638),\n",
       " (765, 0.02092683713211763),\n",
       " (2281, 0.020914819657038775),\n",
       " (4076, 0.020890063840555348),\n",
       " (3799, 0.020884345708652583),\n",
       " (162, 0.02087964269139511),\n",
       " (4627, 0.020858554409346863),\n",
       " (95, 0.020857382429950052),\n",
       " (2352, 0.020846774451591367),\n",
       " (1258, 0.020843080746179833),\n",
       " (1153, 0.020816032788112695),\n",
       " (1278, 0.020800419148734525),\n",
       " (692, 0.020784884983437564),\n",
       " (138, 0.020782607784486422),\n",
       " (2450, 0.020773128912082814),\n",
       " (82, 0.020761692849445748),\n",
       " (4544, 0.02074688394843693),\n",
       " (4634, 0.020743580331079366),\n",
       " (3023, 0.02072988825495914),\n",
       " (1628, 0.020721914721563903),\n",
       " (2435, 0.020707854239679878),\n",
       " (292, 0.020663184865522113),\n",
       " (1741, 0.020661167380136986),\n",
       " (3012, 0.020659868474448093),\n",
       " (4484, 0.020505882148993973),\n",
       " (1166, 0.02049347420348556),\n",
       " (982, 0.02048465397934531),\n",
       " (137, 0.020476955098658666),\n",
       " (2990, 0.020405126162436164),\n",
       " (3551, 0.020380905658752845),\n",
       " (1625, 0.02037128608403978),\n",
       " (1672, 0.02027524566822796),\n",
       " (4179, 0.0202574947616791),\n",
       " (2639, 0.02021824860000264),\n",
       " (4191, 0.0201651343817236),\n",
       " (651, 0.020155439709501774),\n",
       " (2818, 0.020153355963877427),\n",
       " (2458, 0.020133165937387405),\n",
       " (3402, 0.020128212187669567),\n",
       " (496, 0.02012309186131362),\n",
       " (449, 0.02010600829289073),\n",
       " (142, 0.020058512756783067),\n",
       " (1803, 0.0200459603971502),\n",
       " (215, 0.0200383479445473),\n",
       " (2754, 0.020017832723802163),\n",
       " (2348, 0.020001677790323632),\n",
       " (3999, 0.019987212395436063),\n",
       " (917, 0.01996795578384907),\n",
       " (901, 0.019956427691313353),\n",
       " (2285, 0.019955756349836817),\n",
       " (3494, 0.01995566506896362),\n",
       " (1, 0.019951686265704968),\n",
       " (3497, 0.01994163445676279),\n",
       " (318, 0.019915504607229604),\n",
       " (868, 0.019909323234925655),\n",
       " (3833, 0.019907683128444244),\n",
       " (1266, 0.019882563667982783),\n",
       " (1967, 0.01987291414131592),\n",
       " (1090, 0.01986272595462192),\n",
       " (1289, 0.019862526499548526),\n",
       " (3360, 0.01984415879535487),\n",
       " (2046, 0.019839252628059988),\n",
       " (2018, 0.019820458909427026),\n",
       " (4247, 0.01975143290457792),\n",
       " (1529, 0.01973022009133247),\n",
       " (3322, 0.019675586878088888),\n",
       " (3371, 0.01967249970081544),\n",
       " (2253, 0.019669271425426467),\n",
       " (2313, 0.01965285710398931),\n",
       " (964, 0.019642057887103194),\n",
       " (1990, 0.019611891065026373),\n",
       " (276, 0.019569724716278946),\n",
       " (981, 0.019568780602820968),\n",
       " (3143, 0.019558680576936743),\n",
       " (3005, 0.01953731681683741),\n",
       " (3190, 0.019533242468104863),\n",
       " (2680, 0.019481399130677094),\n",
       " (193, 0.019473894039861047),\n",
       " (319, 0.019448559200118034),\n",
       " (4583, 0.0194253981575985),\n",
       " (695, 0.019424941518529776),\n",
       " (3822, 0.01941128569704325),\n",
       " (2745, 0.019390959876378788),\n",
       " (1419, 0.019340396675823462),\n",
       " (2323, 0.01933983482248095),\n",
       " (1126, 0.019323982538060728),\n",
       " (1952, 0.019306385489948466),\n",
       " (2646, 0.019302510042900872),\n",
       " (4116, 0.019291104673119844),\n",
       " (1490, 0.019288779374626662),\n",
       " (4560, 0.019278104846078378),\n",
       " (2048, 0.019268164459933096),\n",
       " (3248, 0.019265395324895223),\n",
       " (257, 0.019262204633820987),\n",
       " (1834, 0.019242629256900515),\n",
       " (2245, 0.01921859873748455),\n",
       " (2051, 0.01920451284321728),\n",
       " (2489, 0.019196899267271535),\n",
       " (2776, 0.019194852610276356),\n",
       " (4086, 0.01916813267804522),\n",
       " (1826, 0.01916089201328691),\n",
       " (3376, 0.019144473585925602),\n",
       " (2433, 0.019111432137197848),\n",
       " (98, 0.019081882049933118),\n",
       " (2131, 0.019081091150954726),\n",
       " (50, 0.01907160213099885),\n",
       " (2971, 0.019039044543474774),\n",
       " (1712, 0.019031689391915504),\n",
       " (1665, 0.018969862855587123),\n",
       " (371, 0.01895477757160393),\n",
       " (2372, 0.018953142100632658),\n",
       " (390, 0.018946995949105212),\n",
       " (921, 0.018938665654526097),\n",
       " (1070, 0.018879125123306743),\n",
       " (812, 0.018795098174592132),\n",
       " (1524, 0.018782868912512334),\n",
       " (4792, 0.01868827074801933),\n",
       " (754, 0.018679871881396283),\n",
       " (3630, 0.018660331390153446),\n",
       " (277, 0.018659185300293875),\n",
       " (1115, 0.01865220236925302),\n",
       " (2592, 0.018644050813020854),\n",
       " (2599, 0.01862627009314169),\n",
       " (366, 0.01860532859957701),\n",
       " (1029, 0.018598047306212406),\n",
       " (263, 0.018569245647190303),\n",
       " (1762, 0.018561670808140147),\n",
       " (4294, 0.018553364004891992),\n",
       " (925, 0.018544370187551916),\n",
       " (3207, 0.018531105545885235),\n",
       " (2695, 0.01849291255211008),\n",
       " (2408, 0.01848872127370436),\n",
       " (784, 0.018449062423707877),\n",
       " (3500, 0.018445281896593904),\n",
       " (1428, 0.0184402276641982),\n",
       " (1231, 0.018409823784719937),\n",
       " (1757, 0.018401747448230574),\n",
       " (1310, 0.018394699494017494),\n",
       " (2229, 0.01838226495688562),\n",
       " (153, 0.018361721803220675),\n",
       " (3386, 0.018359977905817438),\n",
       " (2956, 0.01835115181006729),\n",
       " (4323, 0.01834305603774317),\n",
       " (234, 0.018327291717434153),\n",
       " (1713, 0.018326875661122382),\n",
       " (1379, 0.018302492129574736),\n",
       " (2081, 0.01828651748253515),\n",
       " (1548, 0.018284424058729615),\n",
       " (912, 0.018266786309894794),\n",
       " (789, 0.018256363610611292),\n",
       " (1237, 0.018238512752061772),\n",
       " (606, 0.01821827633342581),\n",
       " (1794, 0.018167698049899844),\n",
       " (1200, 0.01816222606746703),\n",
       " (4037, 0.018138559431527156),\n",
       " (442, 0.01813087819024968),\n",
       " (685, 0.018127995411304384),\n",
       " (3569, 0.018116213136506135),\n",
       " (2100, 0.018103110701477663),\n",
       " (81, 0.018086569622248472),\n",
       " (4232, 0.018064111308281377),\n",
       " (1131, 0.0180584410013842),\n",
       " (4636, 0.018017228350566378),\n",
       " (2790, 0.017978995702343722),\n",
       " (3943, 0.017969448889802134),\n",
       " (463, 0.017961482580876004),\n",
       " (2239, 0.017959952036067813),\n",
       " (1522, 0.017958670682162003),\n",
       " (572, 0.01794295429290468),\n",
       " (3324, 0.01794283866974791),\n",
       " (1297, 0.017937354956166187),\n",
       " (2674, 0.017926906788212744),\n",
       " (115, 0.01791257437283119),\n",
       " (3033, 0.01791134626320439),\n",
       " (1270, 0.017908107440313122),\n",
       " (3146, 0.017904182493562536),\n",
       " (1727, 0.01789980602101184),\n",
       " (4286, 0.01788396689314474),\n",
       " (4626, 0.017881613884532802),\n",
       " (3459, 0.01788159087713727),\n",
       " (3944, 0.0178802460279133),\n",
       " (799, 0.017874962689998587),\n",
       " (875, 0.017871569392481475),\n",
       " (759, 0.017865214170653145),\n",
       " (2389, 0.017798163058767694),\n",
       " (1616, 0.017792845597108017),\n",
       " (4757, 0.017739051275341856),\n",
       " (1254, 0.017724676950631166),\n",
       " (1015, 0.017721966485955573),\n",
       " (2734, 0.01771929004107262),\n",
       " (731, 0.017711596586925088),\n",
       " (1165, 0.017708443357169847),\n",
       " (1465, 0.01770421843031061),\n",
       " (2298, 0.0176866079690459),\n",
       " (3598, 0.017684544777884975),\n",
       " (4668, 0.01767936761521763),\n",
       " (3354, 0.01767645284787189),\n",
       " (1675, 0.017670963715536295),\n",
       " (1331, 0.017665614955056908),\n",
       " (1242, 0.017630161534194694),\n",
       " (625, 0.017585040566881732),\n",
       " (85, 0.01758486590427519),\n",
       " (3907, 0.017572013266792676),\n",
       " (774, 0.01755050769058405),\n",
       " (2394, 0.017550403630932975),\n",
       " (1764, 0.017547918903461267),\n",
       " (286, 0.017542797397268543),\n",
       " (1901, 0.017541371020051655),\n",
       " (2124, 0.01751736372525396),\n",
       " (510, 0.017517148651717794),\n",
       " (1700, 0.017516324310829178),\n",
       " (430, 0.017515169980203197),\n",
       " (4345, 0.017513772458439436),\n",
       " (3013, 0.01751238470003412),\n",
       " ...]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(list(enumerate(cos_mat[0])), reverse=True, key = lambda x: x[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7b3ca8a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_recomm(movie, n):\n",
    "    \n",
    "    # get index from dataframe\n",
    "    index = df[df['title'] ==  movie].index[0]\n",
    "    \n",
    "    # sort top n similar movies\n",
    "    similar_movies = sorted(list(enumerate(cos_mat[index])), reverse=True, key = lambda x: x[1])\n",
    "    \n",
    "    # extract names from dataframe and return the movie names\n",
    "    recomm_mov = []\n",
    "    for i in similar_movies[1:n+1]:\n",
    "        recomm_mov.append(df.iloc[i[0]].title)\n",
    "        \n",
    "    return recomm_mov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "e079eba9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The Helix... Loaded', 'The Book of Life', 'Apollo 18', 'Aliens']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# test the function\n",
    "get_recomm('Avatar',4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "debaa9b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The Dark Knight Rises', 'Batman Begins', 'Batman Returns']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recomm(\"The Dark Knight\", 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "b82d4ff4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Mission: Impossible III',\n",
       " 'Mission: Impossible II',\n",
       " 'Mission: Impossible - Ghost Protocol']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recomm(\"Mission: Impossible\", 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d116258",
   "metadata": {},
   "source": [
    "Up until now, our movie recommendation system has relied solely on movie names. However, there is a limitation: if a movie name is not present in the DataFrame, or what if a user wants to get recommendations based on cast and crews? our system won’t function as intended. But fear not, for we have a solution at hand! Remember how we incorporated the cast and crew information into the corpus? Well, it’s time to leverage that.\n",
    "\n",
    "Although the cast and crew details are not directly included in our DataFrame, we can still make use of them. Here’s where TF-IDF comes to the rescue. By applying TF-IDF transformation to the keywords or tags, we can convert them into vectors of the same length as our cosine matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bfce877b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Christopher Nolan'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywords = \"Christopher Nolan\"\n",
    "keywords = keywords.split()\n",
    "keywords = \" \".join(keywords)\n",
    "keywords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "a5b00f0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x30258 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 2 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# transform the string to vector representation\n",
    "key_tfidf = tfidf.transform([keywords])\n",
    "key_tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "90aaad79",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 30258)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "key_tfidf.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "db8cc002",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute cosine similarity    \n",
    "result = cosine_similarity(key_tfidf, tfidf_matrix)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "5ffac300",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1196, 0.24981625443638547),\n",
       " (1033, 0.2269891672477714),\n",
       " (14, 0.2173854313908544),\n",
       " (119, 0.20393152996372776),\n",
       " (95, 0.2003657551723531),\n",
       " (1393, 0.1974346497892047),\n",
       " (96, 0.19031466183323786),\n",
       " (65, 0.1674131953333691),\n",
       " (196, 0.16184034862758323),\n",
       " (4644, 0.15860445527894698),\n",
       " (3463, 0.15540336731717094),\n",
       " (3, 0.154612547194409),\n",
       " (3572, 0.14683765801039356),\n",
       " (2868, 0.13499580580114212),\n",
       " (106, 0.13381206120830644),\n",
       " (3577, 0.13307587784510655),\n",
       " (423, 0.12305006734967193),\n",
       " (4484, 0.1191808653725676),\n",
       " (3694, 0.10018861343708382),\n",
       " (3769, 0.10010832362964717),\n",
       " (1758, 0.09940750004125574),\n",
       " (3551, 0.09652012833828355),\n",
       " (2199, 0.09564541105303324),\n",
       " (584, 0.0931283399329479),\n",
       " (1650, 0.09261229846062927),\n",
       " (1658, 0.0879336104268975),\n",
       " (1802, 0.0833693181712317),\n",
       " (2388, 0.07681298891404421),\n",
       " (3235, 0.07551090003440986),\n",
       " (1649, 0.07523808200620612),\n",
       " (2745, 0.07227935010452113),\n",
       " (1109, 0.07185852011805015),\n",
       " (506, 0.07178809112789503),\n",
       " (2028, 0.07137891631069808),\n",
       " (4605, 0.07119388729009309),\n",
       " (4600, 0.07076249167948812),\n",
       " (2116, 0.0704016767086314),\n",
       " (4336, 0.06923229874989566),\n",
       " (2505, 0.06871095420411844),\n",
       " (1873, 0.06860253526917737),\n",
       " (504, 0.06702038360133694),\n",
       " (2721, 0.06682313375358252),\n",
       " (2300, 0.06656683278613797),\n",
       " (2640, 0.06543983904648867),\n",
       " (2490, 0.06536957976962254),\n",
       " (3013, 0.06527700500482946),\n",
       " (179, 0.06475200786734539),\n",
       " (721, 0.06472004415634831),\n",
       " (2735, 0.06463701848200568),\n",
       " (2212, 0.06414597635721292),\n",
       " (4712, 0.06351516129517121),\n",
       " (1875, 0.06307133108368049),\n",
       " (3086, 0.06292854519394273),\n",
       " (2800, 0.06282180369838103),\n",
       " (2685, 0.06280236332564001),\n",
       " (1720, 0.06200039692849215),\n",
       " (620, 0.061954632241860236),\n",
       " (1204, 0.06109563283870781),\n",
       " (3782, 0.06108832292295993),\n",
       " (3568, 0.06088272427318363),\n",
       " (3880, 0.06084228180332755),\n",
       " (134, 0.060685432240044176),\n",
       " (2493, 0.060647903537160194),\n",
       " (546, 0.06059675264378567),\n",
       " (1898, 0.06054522088465814),\n",
       " (966, 0.060227559076534994),\n",
       " (2901, 0.06004224948074856),\n",
       " (2946, 0.05984759960403913),\n",
       " (3558, 0.059836363970136905),\n",
       " (744, 0.05980749042367299),\n",
       " (1648, 0.05936823249577788),\n",
       " (2536, 0.05905534956445207),\n",
       " (2902, 0.05874620684386149),\n",
       " (2256, 0.05870994693094007),\n",
       " (308, 0.05833031930201465),\n",
       " (1414, 0.058307789241616996),\n",
       " (3000, 0.05820465957385059),\n",
       " (4004, 0.05807541411051724),\n",
       " (2178, 0.05779896582819582),\n",
       " (2128, 0.05748667705819411),\n",
       " (2392, 0.05748567528964157),\n",
       " (3179, 0.057058268151358206),\n",
       " (883, 0.056737890838697694),\n",
       " (977, 0.05657509709633583),\n",
       " (3350, 0.0559302723410003),\n",
       " (614, 0.055582499880808094),\n",
       " (2971, 0.05550697441998932),\n",
       " (813, 0.05536829861533532),\n",
       " (4282, 0.055113540643762726),\n",
       " (416, 0.05511340957285675),\n",
       " (4351, 0.054992040766365505),\n",
       " (4380, 0.05497541016318751),\n",
       " (3370, 0.054959576426669945),\n",
       " (3239, 0.05482901861535016),\n",
       " (3564, 0.05449382971978371),\n",
       " (2646, 0.054375081033515654),\n",
       " (1703, 0.05436672370355733),\n",
       " (4211, 0.05434233949139726),\n",
       " (4475, 0.05419233886302158),\n",
       " (2285, 0.054029036507473926),\n",
       " (516, 0.05378957682823205),\n",
       " (66, 0.053751378614768135),\n",
       " (1428, 0.053587606588144814),\n",
       " (4149, 0.053489933390791986),\n",
       " (1894, 0.05337593709797608),\n",
       " (1722, 0.053352603242626806),\n",
       " (442, 0.05268863190030039),\n",
       " (1614, 0.05257984936412062),\n",
       " (1586, 0.05252459317769669),\n",
       " (4439, 0.05231399633596298),\n",
       " (2402, 0.05184860199852499),\n",
       " (2571, 0.05137217392808276),\n",
       " (870, 0.050949748845816774),\n",
       " (3458, 0.05090767544297724),\n",
       " (3248, 0.05066814986603585),\n",
       " (2988, 0.05035022148465466),\n",
       " (1617, 0.050192443160401896),\n",
       " (752, 0.04995935186442727),\n",
       " (2867, 0.04989289494157647),\n",
       " (360, 0.04986149469450606),\n",
       " (2707, 0.049734882163823206),\n",
       " (1620, 0.049695666144272656),\n",
       " (4332, 0.04934561765967976),\n",
       " (4106, 0.049309144037639516),\n",
       " (4020, 0.049008237566853304),\n",
       " (1119, 0.048850047285407164),\n",
       " (918, 0.04828612947650135),\n",
       " (4339, 0.048283254278378596),\n",
       " (1296, 0.04820268505477113),\n",
       " (1165, 0.0479445687681433),\n",
       " (1591, 0.04791878811847267),\n",
       " (1135, 0.047541872740818746),\n",
       " (3918, 0.04684423302252255),\n",
       " (347, 0.046475908328901375),\n",
       " (1152, 0.046466923452405565),\n",
       " (3512, 0.046173133032687524),\n",
       " (3107, 0.04607716289337567),\n",
       " (2764, 0.0457903160675824),\n",
       " (1740, 0.0456909366378608),\n",
       " (701, 0.045644065236246256),\n",
       " (2211, 0.04558950928415923),\n",
       " (2360, 0.04558029876573165),\n",
       " (4718, 0.04547862509626085),\n",
       " (1208, 0.045275034210313966),\n",
       " (2250, 0.045272421538858944),\n",
       " (402, 0.04522847206077858),\n",
       " (3095, 0.0451190244522106),\n",
       " (3113, 0.04491106031938951),\n",
       " (1909, 0.04471661607829449),\n",
       " (824, 0.04460195168292522),\n",
       " (2100, 0.04450324133320751),\n",
       " (450, 0.0444409697398483),\n",
       " (2367, 0.04416725538948911),\n",
       " (1549, 0.044035297654780675),\n",
       " (312, 0.0439791185644837),\n",
       " (4625, 0.043628544367490835),\n",
       " (4478, 0.04355458459887013),\n",
       " (2433, 0.042928892921909996),\n",
       " (822, 0.04284930092450142),\n",
       " (4623, 0.04257349358754398),\n",
       " (1954, 0.04255013766272219),\n",
       " (184, 0.04228755955287722),\n",
       " (874, 0.041768915819700256),\n",
       " (354, 0.04153236938213842),\n",
       " (3874, 0.04133730494707753),\n",
       " (911, 0.041327910890465155),\n",
       " (3057, 0.041196070231660435),\n",
       " (1464, 0.04094165382175711),\n",
       " (2832, 0.0409162248733746),\n",
       " (3724, 0.04012735639692053),\n",
       " (2357, 0.04000044135037787),\n",
       " (4708, 0.03980084660065955),\n",
       " (493, 0.039445543207659396),\n",
       " (4113, 0.03927850411897486),\n",
       " (3499, 0.03846438185245444),\n",
       " (4715, 0.03825621593289618),\n",
       " (1888, 0.03785000854070863),\n",
       " (428, 0.03779057157006583),\n",
       " (2592, 0.03742659793950221),\n",
       " (481, 0.03740804707721603),\n",
       " (318, 0.03727162833844025),\n",
       " (3404, 0.03627222144316417),\n",
       " (285, 0.035274692605558676),\n",
       " (3372, 0.035099346520831125),\n",
       " (717, 0.03453248405731712),\n",
       " (3184, 0.03437701009342057),\n",
       " (3516, 0.03368225348877557),\n",
       " (2955, 0.03286822059996482),\n",
       " (1813, 0.03278035831435198),\n",
       " (2768, 0.031541468543803186),\n",
       " (1485, 0.03019178064350069),\n",
       " (0, 0.0),\n",
       " (1, 0.0),\n",
       " (2, 0.0),\n",
       " (4, 0.0),\n",
       " (5, 0.0),\n",
       " (6, 0.0),\n",
       " (7, 0.0),\n",
       " (8, 0.0),\n",
       " (9, 0.0),\n",
       " (10, 0.0),\n",
       " (11, 0.0),\n",
       " (12, 0.0),\n",
       " (13, 0.0),\n",
       " (15, 0.0),\n",
       " (16, 0.0),\n",
       " (17, 0.0),\n",
       " (18, 0.0),\n",
       " (19, 0.0),\n",
       " (20, 0.0),\n",
       " (21, 0.0),\n",
       " (22, 0.0),\n",
       " (23, 0.0),\n",
       " (24, 0.0),\n",
       " (25, 0.0),\n",
       " (26, 0.0),\n",
       " (27, 0.0),\n",
       " (28, 0.0),\n",
       " (29, 0.0),\n",
       " (30, 0.0),\n",
       " (31, 0.0),\n",
       " (32, 0.0),\n",
       " (33, 0.0),\n",
       " (34, 0.0),\n",
       " (35, 0.0),\n",
       " (36, 0.0),\n",
       " (37, 0.0),\n",
       " (38, 0.0),\n",
       " (39, 0.0),\n",
       " (40, 0.0),\n",
       " (41, 0.0),\n",
       " (42, 0.0),\n",
       " (43, 0.0),\n",
       " (44, 0.0),\n",
       " (45, 0.0),\n",
       " (46, 0.0),\n",
       " (47, 0.0),\n",
       " (48, 0.0),\n",
       " (49, 0.0),\n",
       " (50, 0.0),\n",
       " (51, 0.0),\n",
       " (52, 0.0),\n",
       " (53, 0.0),\n",
       " (54, 0.0),\n",
       " (55, 0.0),\n",
       " (56, 0.0),\n",
       " (57, 0.0),\n",
       " (58, 0.0),\n",
       " (59, 0.0),\n",
       " (60, 0.0),\n",
       " (61, 0.0),\n",
       " (62, 0.0),\n",
       " (63, 0.0),\n",
       " (64, 0.0),\n",
       " (67, 0.0),\n",
       " (68, 0.0),\n",
       " (69, 0.0),\n",
       " (70, 0.0),\n",
       " (71, 0.0),\n",
       " (72, 0.0),\n",
       " (73, 0.0),\n",
       " (74, 0.0),\n",
       " (75, 0.0),\n",
       " (76, 0.0),\n",
       " (77, 0.0),\n",
       " (78, 0.0),\n",
       " (79, 0.0),\n",
       " (80, 0.0),\n",
       " (81, 0.0),\n",
       " (82, 0.0),\n",
       " (83, 0.0),\n",
       " (84, 0.0),\n",
       " (85, 0.0),\n",
       " (86, 0.0),\n",
       " (87, 0.0),\n",
       " (88, 0.0),\n",
       " (89, 0.0),\n",
       " (90, 0.0),\n",
       " (91, 0.0),\n",
       " (92, 0.0),\n",
       " (93, 0.0),\n",
       " (94, 0.0),\n",
       " (97, 0.0),\n",
       " (98, 0.0),\n",
       " (99, 0.0),\n",
       " (100, 0.0),\n",
       " (101, 0.0),\n",
       " (102, 0.0),\n",
       " (103, 0.0),\n",
       " (104, 0.0),\n",
       " (105, 0.0),\n",
       " (107, 0.0),\n",
       " (108, 0.0),\n",
       " (109, 0.0),\n",
       " (110, 0.0),\n",
       " (111, 0.0),\n",
       " (112, 0.0),\n",
       " (113, 0.0),\n",
       " (114, 0.0),\n",
       " (115, 0.0),\n",
       " (116, 0.0),\n",
       " (117, 0.0),\n",
       " (118, 0.0),\n",
       " (120, 0.0),\n",
       " (121, 0.0),\n",
       " (122, 0.0),\n",
       " (123, 0.0),\n",
       " (124, 0.0),\n",
       " (125, 0.0),\n",
       " (126, 0.0),\n",
       " (127, 0.0),\n",
       " (128, 0.0),\n",
       " (129, 0.0),\n",
       " (130, 0.0),\n",
       " (131, 0.0),\n",
       " (132, 0.0),\n",
       " (133, 0.0),\n",
       " (135, 0.0),\n",
       " (136, 0.0),\n",
       " (137, 0.0),\n",
       " (138, 0.0),\n",
       " (139, 0.0),\n",
       " (140, 0.0),\n",
       " (141, 0.0),\n",
       " (142, 0.0),\n",
       " (143, 0.0),\n",
       " (144, 0.0),\n",
       " (145, 0.0),\n",
       " (146, 0.0),\n",
       " (147, 0.0),\n",
       " (148, 0.0),\n",
       " (149, 0.0),\n",
       " (150, 0.0),\n",
       " (151, 0.0),\n",
       " (152, 0.0),\n",
       " (153, 0.0),\n",
       " (154, 0.0),\n",
       " (155, 0.0),\n",
       " (156, 0.0),\n",
       " (157, 0.0),\n",
       " (158, 0.0),\n",
       " (159, 0.0),\n",
       " (160, 0.0),\n",
       " (161, 0.0),\n",
       " (162, 0.0),\n",
       " (163, 0.0),\n",
       " (164, 0.0),\n",
       " (165, 0.0),\n",
       " (166, 0.0),\n",
       " (167, 0.0),\n",
       " (168, 0.0),\n",
       " (169, 0.0),\n",
       " (170, 0.0),\n",
       " (171, 0.0),\n",
       " (172, 0.0),\n",
       " (173, 0.0),\n",
       " (174, 0.0),\n",
       " (175, 0.0),\n",
       " (176, 0.0),\n",
       " (177, 0.0),\n",
       " (178, 0.0),\n",
       " (180, 0.0),\n",
       " (181, 0.0),\n",
       " (182, 0.0),\n",
       " (183, 0.0),\n",
       " (185, 0.0),\n",
       " (186, 0.0),\n",
       " (187, 0.0),\n",
       " (188, 0.0),\n",
       " (189, 0.0),\n",
       " (190, 0.0),\n",
       " (191, 0.0),\n",
       " (192, 0.0),\n",
       " (193, 0.0),\n",
       " (194, 0.0),\n",
       " (195, 0.0),\n",
       " (197, 0.0),\n",
       " (198, 0.0),\n",
       " (199, 0.0),\n",
       " (200, 0.0),\n",
       " (201, 0.0),\n",
       " (202, 0.0),\n",
       " (203, 0.0),\n",
       " (204, 0.0),\n",
       " (205, 0.0),\n",
       " (206, 0.0),\n",
       " (207, 0.0),\n",
       " (208, 0.0),\n",
       " (209, 0.0),\n",
       " (210, 0.0),\n",
       " (211, 0.0),\n",
       " (212, 0.0),\n",
       " (213, 0.0),\n",
       " (214, 0.0),\n",
       " (215, 0.0),\n",
       " (216, 0.0),\n",
       " (217, 0.0),\n",
       " (218, 0.0),\n",
       " (219, 0.0),\n",
       " (220, 0.0),\n",
       " (221, 0.0),\n",
       " (222, 0.0),\n",
       " (223, 0.0),\n",
       " (224, 0.0),\n",
       " (225, 0.0),\n",
       " (226, 0.0),\n",
       " (227, 0.0),\n",
       " (228, 0.0),\n",
       " (229, 0.0),\n",
       " (230, 0.0),\n",
       " (231, 0.0),\n",
       " (232, 0.0),\n",
       " (233, 0.0),\n",
       " (234, 0.0),\n",
       " (235, 0.0),\n",
       " (236, 0.0),\n",
       " (237, 0.0),\n",
       " (238, 0.0),\n",
       " (239, 0.0),\n",
       " (240, 0.0),\n",
       " (241, 0.0),\n",
       " (242, 0.0),\n",
       " (243, 0.0),\n",
       " (244, 0.0),\n",
       " (245, 0.0),\n",
       " (246, 0.0),\n",
       " (247, 0.0),\n",
       " (248, 0.0),\n",
       " (249, 0.0),\n",
       " (250, 0.0),\n",
       " (251, 0.0),\n",
       " (252, 0.0),\n",
       " (253, 0.0),\n",
       " (254, 0.0),\n",
       " (255, 0.0),\n",
       " (256, 0.0),\n",
       " (257, 0.0),\n",
       " (258, 0.0),\n",
       " (259, 0.0),\n",
       " (260, 0.0),\n",
       " (261, 0.0),\n",
       " (262, 0.0),\n",
       " (263, 0.0),\n",
       " (264, 0.0),\n",
       " (265, 0.0),\n",
       " (266, 0.0),\n",
       " (267, 0.0),\n",
       " (268, 0.0),\n",
       " (269, 0.0),\n",
       " (270, 0.0),\n",
       " (271, 0.0),\n",
       " (272, 0.0),\n",
       " (273, 0.0),\n",
       " (274, 0.0),\n",
       " (275, 0.0),\n",
       " (276, 0.0),\n",
       " (277, 0.0),\n",
       " (278, 0.0),\n",
       " (279, 0.0),\n",
       " (280, 0.0),\n",
       " (281, 0.0),\n",
       " (282, 0.0),\n",
       " (283, 0.0),\n",
       " (284, 0.0),\n",
       " (286, 0.0),\n",
       " (287, 0.0),\n",
       " (288, 0.0),\n",
       " (289, 0.0),\n",
       " (290, 0.0),\n",
       " (291, 0.0),\n",
       " (292, 0.0),\n",
       " (293, 0.0),\n",
       " (294, 0.0),\n",
       " (295, 0.0),\n",
       " (296, 0.0),\n",
       " (297, 0.0),\n",
       " (298, 0.0),\n",
       " (299, 0.0),\n",
       " (300, 0.0),\n",
       " (301, 0.0),\n",
       " (302, 0.0),\n",
       " (303, 0.0),\n",
       " (304, 0.0),\n",
       " (305, 0.0),\n",
       " (306, 0.0),\n",
       " (307, 0.0),\n",
       " (309, 0.0),\n",
       " (310, 0.0),\n",
       " (311, 0.0),\n",
       " (313, 0.0),\n",
       " (314, 0.0),\n",
       " (315, 0.0),\n",
       " (316, 0.0),\n",
       " (317, 0.0),\n",
       " (319, 0.0),\n",
       " (320, 0.0),\n",
       " (321, 0.0),\n",
       " (322, 0.0),\n",
       " (323, 0.0),\n",
       " (324, 0.0),\n",
       " (325, 0.0),\n",
       " (326, 0.0),\n",
       " (327, 0.0),\n",
       " (328, 0.0),\n",
       " (329, 0.0),\n",
       " (330, 0.0),\n",
       " (331, 0.0),\n",
       " (332, 0.0),\n",
       " (333, 0.0),\n",
       " (334, 0.0),\n",
       " (335, 0.0),\n",
       " (336, 0.0),\n",
       " (337, 0.0),\n",
       " (338, 0.0),\n",
       " (339, 0.0),\n",
       " (340, 0.0),\n",
       " (341, 0.0),\n",
       " (342, 0.0),\n",
       " (343, 0.0),\n",
       " (344, 0.0),\n",
       " (345, 0.0),\n",
       " (346, 0.0),\n",
       " (348, 0.0),\n",
       " (349, 0.0),\n",
       " (350, 0.0),\n",
       " (351, 0.0),\n",
       " (352, 0.0),\n",
       " (353, 0.0),\n",
       " (355, 0.0),\n",
       " (356, 0.0),\n",
       " (357, 0.0),\n",
       " (358, 0.0),\n",
       " (359, 0.0),\n",
       " (361, 0.0),\n",
       " (362, 0.0),\n",
       " (363, 0.0),\n",
       " (364, 0.0),\n",
       " (365, 0.0),\n",
       " (366, 0.0),\n",
       " (367, 0.0),\n",
       " (368, 0.0),\n",
       " (369, 0.0),\n",
       " (370, 0.0),\n",
       " (371, 0.0),\n",
       " (372, 0.0),\n",
       " (373, 0.0),\n",
       " (374, 0.0),\n",
       " (375, 0.0),\n",
       " (376, 0.0),\n",
       " (377, 0.0),\n",
       " (378, 0.0),\n",
       " (379, 0.0),\n",
       " (380, 0.0),\n",
       " (381, 0.0),\n",
       " (382, 0.0),\n",
       " (383, 0.0),\n",
       " (384, 0.0),\n",
       " (385, 0.0),\n",
       " (386, 0.0),\n",
       " (387, 0.0),\n",
       " (388, 0.0),\n",
       " (389, 0.0),\n",
       " (390, 0.0),\n",
       " (391, 0.0),\n",
       " (392, 0.0),\n",
       " (393, 0.0),\n",
       " (394, 0.0),\n",
       " (395, 0.0),\n",
       " (396, 0.0),\n",
       " (397, 0.0),\n",
       " (398, 0.0),\n",
       " (399, 0.0),\n",
       " (400, 0.0),\n",
       " (401, 0.0),\n",
       " (403, 0.0),\n",
       " (404, 0.0),\n",
       " (405, 0.0),\n",
       " (406, 0.0),\n",
       " (407, 0.0),\n",
       " (408, 0.0),\n",
       " (409, 0.0),\n",
       " (410, 0.0),\n",
       " (411, 0.0),\n",
       " (412, 0.0),\n",
       " (413, 0.0),\n",
       " (414, 0.0),\n",
       " (415, 0.0),\n",
       " (417, 0.0),\n",
       " (418, 0.0),\n",
       " (419, 0.0),\n",
       " (420, 0.0),\n",
       " (421, 0.0),\n",
       " (422, 0.0),\n",
       " (424, 0.0),\n",
       " (425, 0.0),\n",
       " (426, 0.0),\n",
       " (427, 0.0),\n",
       " (429, 0.0),\n",
       " (430, 0.0),\n",
       " (431, 0.0),\n",
       " (432, 0.0),\n",
       " (433, 0.0),\n",
       " (434, 0.0),\n",
       " (435, 0.0),\n",
       " (436, 0.0),\n",
       " (437, 0.0),\n",
       " (438, 0.0),\n",
       " (439, 0.0),\n",
       " (440, 0.0),\n",
       " (441, 0.0),\n",
       " (443, 0.0),\n",
       " (444, 0.0),\n",
       " (445, 0.0),\n",
       " (446, 0.0),\n",
       " (447, 0.0),\n",
       " (448, 0.0),\n",
       " (449, 0.0),\n",
       " (451, 0.0),\n",
       " (452, 0.0),\n",
       " (453, 0.0),\n",
       " (454, 0.0),\n",
       " (455, 0.0),\n",
       " (456, 0.0),\n",
       " (457, 0.0),\n",
       " (458, 0.0),\n",
       " (459, 0.0),\n",
       " (460, 0.0),\n",
       " (461, 0.0),\n",
       " (462, 0.0),\n",
       " (463, 0.0),\n",
       " (464, 0.0),\n",
       " (465, 0.0),\n",
       " (466, 0.0),\n",
       " (467, 0.0),\n",
       " (468, 0.0),\n",
       " (469, 0.0),\n",
       " (470, 0.0),\n",
       " (471, 0.0),\n",
       " (472, 0.0),\n",
       " (473, 0.0),\n",
       " (474, 0.0),\n",
       " (475, 0.0),\n",
       " (476, 0.0),\n",
       " (477, 0.0),\n",
       " (478, 0.0),\n",
       " (479, 0.0),\n",
       " (480, 0.0),\n",
       " (482, 0.0),\n",
       " (483, 0.0),\n",
       " (484, 0.0),\n",
       " (485, 0.0),\n",
       " (486, 0.0),\n",
       " (487, 0.0),\n",
       " (488, 0.0),\n",
       " (489, 0.0),\n",
       " (490, 0.0),\n",
       " (491, 0.0),\n",
       " (492, 0.0),\n",
       " (494, 0.0),\n",
       " (495, 0.0),\n",
       " (496, 0.0),\n",
       " (497, 0.0),\n",
       " (498, 0.0),\n",
       " (499, 0.0),\n",
       " (500, 0.0),\n",
       " (501, 0.0),\n",
       " (502, 0.0),\n",
       " (503, 0.0),\n",
       " (505, 0.0),\n",
       " (507, 0.0),\n",
       " (508, 0.0),\n",
       " (509, 0.0),\n",
       " (510, 0.0),\n",
       " (511, 0.0),\n",
       " (512, 0.0),\n",
       " (513, 0.0),\n",
       " (514, 0.0),\n",
       " (515, 0.0),\n",
       " (517, 0.0),\n",
       " (518, 0.0),\n",
       " (519, 0.0),\n",
       " (520, 0.0),\n",
       " (521, 0.0),\n",
       " (522, 0.0),\n",
       " (523, 0.0),\n",
       " (524, 0.0),\n",
       " (525, 0.0),\n",
       " (526, 0.0),\n",
       " (527, 0.0),\n",
       " (528, 0.0),\n",
       " (529, 0.0),\n",
       " (530, 0.0),\n",
       " (531, 0.0),\n",
       " (532, 0.0),\n",
       " (533, 0.0),\n",
       " (534, 0.0),\n",
       " (535, 0.0),\n",
       " (536, 0.0),\n",
       " (537, 0.0),\n",
       " (538, 0.0),\n",
       " (539, 0.0),\n",
       " (540, 0.0),\n",
       " (541, 0.0),\n",
       " (542, 0.0),\n",
       " (543, 0.0),\n",
       " (544, 0.0),\n",
       " (545, 0.0),\n",
       " (547, 0.0),\n",
       " (548, 0.0),\n",
       " (549, 0.0),\n",
       " (550, 0.0),\n",
       " (551, 0.0),\n",
       " (552, 0.0),\n",
       " (553, 0.0),\n",
       " (554, 0.0),\n",
       " (555, 0.0),\n",
       " (556, 0.0),\n",
       " (557, 0.0),\n",
       " (558, 0.0),\n",
       " (559, 0.0),\n",
       " (560, 0.0),\n",
       " (561, 0.0),\n",
       " (562, 0.0),\n",
       " (563, 0.0),\n",
       " (564, 0.0),\n",
       " (565, 0.0),\n",
       " (566, 0.0),\n",
       " (567, 0.0),\n",
       " (568, 0.0),\n",
       " (569, 0.0),\n",
       " (570, 0.0),\n",
       " (571, 0.0),\n",
       " (572, 0.0),\n",
       " (573, 0.0),\n",
       " (574, 0.0),\n",
       " (575, 0.0),\n",
       " (576, 0.0),\n",
       " (577, 0.0),\n",
       " (578, 0.0),\n",
       " (579, 0.0),\n",
       " (580, 0.0),\n",
       " (581, 0.0),\n",
       " (582, 0.0),\n",
       " (583, 0.0),\n",
       " (585, 0.0),\n",
       " (586, 0.0),\n",
       " (587, 0.0),\n",
       " (588, 0.0),\n",
       " (589, 0.0),\n",
       " (590, 0.0),\n",
       " (591, 0.0),\n",
       " (592, 0.0),\n",
       " (593, 0.0),\n",
       " (594, 0.0),\n",
       " (595, 0.0),\n",
       " (596, 0.0),\n",
       " (597, 0.0),\n",
       " (598, 0.0),\n",
       " (599, 0.0),\n",
       " (600, 0.0),\n",
       " (601, 0.0),\n",
       " (602, 0.0),\n",
       " (603, 0.0),\n",
       " (604, 0.0),\n",
       " (605, 0.0),\n",
       " (606, 0.0),\n",
       " (607, 0.0),\n",
       " (608, 0.0),\n",
       " (609, 0.0),\n",
       " (610, 0.0),\n",
       " (611, 0.0),\n",
       " (612, 0.0),\n",
       " (613, 0.0),\n",
       " (615, 0.0),\n",
       " (616, 0.0),\n",
       " (617, 0.0),\n",
       " (618, 0.0),\n",
       " (619, 0.0),\n",
       " (621, 0.0),\n",
       " (622, 0.0),\n",
       " (623, 0.0),\n",
       " (624, 0.0),\n",
       " (625, 0.0),\n",
       " (626, 0.0),\n",
       " (627, 0.0),\n",
       " (628, 0.0),\n",
       " (629, 0.0),\n",
       " (630, 0.0),\n",
       " (631, 0.0),\n",
       " (632, 0.0),\n",
       " (633, 0.0),\n",
       " (634, 0.0),\n",
       " (635, 0.0),\n",
       " (636, 0.0),\n",
       " (637, 0.0),\n",
       " (638, 0.0),\n",
       " (639, 0.0),\n",
       " (640, 0.0),\n",
       " (641, 0.0),\n",
       " (642, 0.0),\n",
       " (643, 0.0),\n",
       " (644, 0.0),\n",
       " (645, 0.0),\n",
       " (646, 0.0),\n",
       " (647, 0.0),\n",
       " (648, 0.0),\n",
       " (649, 0.0),\n",
       " (650, 0.0),\n",
       " (651, 0.0),\n",
       " (652, 0.0),\n",
       " (653, 0.0),\n",
       " (654, 0.0),\n",
       " (655, 0.0),\n",
       " (656, 0.0),\n",
       " (657, 0.0),\n",
       " (658, 0.0),\n",
       " (659, 0.0),\n",
       " (660, 0.0),\n",
       " (661, 0.0),\n",
       " (662, 0.0),\n",
       " (663, 0.0),\n",
       " (664, 0.0),\n",
       " (665, 0.0),\n",
       " (666, 0.0),\n",
       " (667, 0.0),\n",
       " (668, 0.0),\n",
       " (669, 0.0),\n",
       " (670, 0.0),\n",
       " (671, 0.0),\n",
       " (672, 0.0),\n",
       " (673, 0.0),\n",
       " (674, 0.0),\n",
       " (675, 0.0),\n",
       " (676, 0.0),\n",
       " (677, 0.0),\n",
       " (678, 0.0),\n",
       " (679, 0.0),\n",
       " (680, 0.0),\n",
       " (681, 0.0),\n",
       " (682, 0.0),\n",
       " (683, 0.0),\n",
       " (684, 0.0),\n",
       " (685, 0.0),\n",
       " (686, 0.0),\n",
       " (687, 0.0),\n",
       " (688, 0.0),\n",
       " (689, 0.0),\n",
       " (690, 0.0),\n",
       " (691, 0.0),\n",
       " (692, 0.0),\n",
       " (693, 0.0),\n",
       " (694, 0.0),\n",
       " (695, 0.0),\n",
       " (696, 0.0),\n",
       " (697, 0.0),\n",
       " (698, 0.0),\n",
       " (699, 0.0),\n",
       " (700, 0.0),\n",
       " (702, 0.0),\n",
       " (703, 0.0),\n",
       " (704, 0.0),\n",
       " (705, 0.0),\n",
       " (706, 0.0),\n",
       " (707, 0.0),\n",
       " (708, 0.0),\n",
       " (709, 0.0),\n",
       " (710, 0.0),\n",
       " (711, 0.0),\n",
       " (712, 0.0),\n",
       " (713, 0.0),\n",
       " (714, 0.0),\n",
       " (715, 0.0),\n",
       " (716, 0.0),\n",
       " (718, 0.0),\n",
       " (719, 0.0),\n",
       " (720, 0.0),\n",
       " (722, 0.0),\n",
       " (723, 0.0),\n",
       " (724, 0.0),\n",
       " (725, 0.0),\n",
       " (726, 0.0),\n",
       " (727, 0.0),\n",
       " (728, 0.0),\n",
       " (729, 0.0),\n",
       " (730, 0.0),\n",
       " (731, 0.0),\n",
       " (732, 0.0),\n",
       " (733, 0.0),\n",
       " (734, 0.0),\n",
       " (735, 0.0),\n",
       " (736, 0.0),\n",
       " (737, 0.0),\n",
       " (738, 0.0),\n",
       " (739, 0.0),\n",
       " (740, 0.0),\n",
       " (741, 0.0),\n",
       " (742, 0.0),\n",
       " (743, 0.0),\n",
       " (745, 0.0),\n",
       " (746, 0.0),\n",
       " (747, 0.0),\n",
       " (748, 0.0),\n",
       " (749, 0.0),\n",
       " (750, 0.0),\n",
       " (751, 0.0),\n",
       " (753, 0.0),\n",
       " (754, 0.0),\n",
       " (755, 0.0),\n",
       " (756, 0.0),\n",
       " (757, 0.0),\n",
       " (758, 0.0),\n",
       " (759, 0.0),\n",
       " (760, 0.0),\n",
       " (761, 0.0),\n",
       " (762, 0.0),\n",
       " (763, 0.0),\n",
       " (764, 0.0),\n",
       " (765, 0.0),\n",
       " (766, 0.0),\n",
       " (767, 0.0),\n",
       " (768, 0.0),\n",
       " (769, 0.0),\n",
       " (770, 0.0),\n",
       " (771, 0.0),\n",
       " (772, 0.0),\n",
       " (773, 0.0),\n",
       " (774, 0.0),\n",
       " (775, 0.0),\n",
       " (776, 0.0),\n",
       " (777, 0.0),\n",
       " (778, 0.0),\n",
       " (779, 0.0),\n",
       " (780, 0.0),\n",
       " (781, 0.0),\n",
       " (782, 0.0),\n",
       " (783, 0.0),\n",
       " (784, 0.0),\n",
       " (785, 0.0),\n",
       " (786, 0.0),\n",
       " (787, 0.0),\n",
       " (788, 0.0),\n",
       " (789, 0.0),\n",
       " (790, 0.0),\n",
       " (791, 0.0),\n",
       " (792, 0.0),\n",
       " (793, 0.0),\n",
       " (794, 0.0),\n",
       " (795, 0.0),\n",
       " (796, 0.0),\n",
       " (797, 0.0),\n",
       " (798, 0.0),\n",
       " (799, 0.0),\n",
       " (800, 0.0),\n",
       " (801, 0.0),\n",
       " (802, 0.0),\n",
       " (803, 0.0),\n",
       " (804, 0.0),\n",
       " (805, 0.0),\n",
       " (806, 0.0),\n",
       " (807, 0.0),\n",
       " (808, 0.0),\n",
       " (809, 0.0),\n",
       " (810, 0.0),\n",
       " (811, 0.0),\n",
       " (812, 0.0),\n",
       " (814, 0.0),\n",
       " (815, 0.0),\n",
       " (816, 0.0),\n",
       " (817, 0.0),\n",
       " (818, 0.0),\n",
       " (819, 0.0),\n",
       " (820, 0.0),\n",
       " (821, 0.0),\n",
       " (823, 0.0),\n",
       " (825, 0.0),\n",
       " (826, 0.0),\n",
       " (827, 0.0),\n",
       " (828, 0.0),\n",
       " (829, 0.0),\n",
       " (830, 0.0),\n",
       " (831, 0.0),\n",
       " (832, 0.0),\n",
       " (833, 0.0),\n",
       " (834, 0.0),\n",
       " (835, 0.0),\n",
       " (836, 0.0),\n",
       " (837, 0.0),\n",
       " (838, 0.0),\n",
       " (839, 0.0),\n",
       " (840, 0.0),\n",
       " (841, 0.0),\n",
       " (842, 0.0),\n",
       " (843, 0.0),\n",
       " (844, 0.0),\n",
       " (845, 0.0),\n",
       " (846, 0.0),\n",
       " (847, 0.0),\n",
       " (848, 0.0),\n",
       " (849, 0.0),\n",
       " (850, 0.0),\n",
       " ...]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# sort top n similar movies   \n",
    "similar_key_movies = sorted(list(enumerate(result[0])), reverse=True, key=lambda x: x[1])\n",
    "similar_key_movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "55ec3011",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_recomm_keywords(keywords, n):\n",
    "    \n",
    "    keywords = keywords.split()\n",
    "    keywords = \" \".join(keywords)\n",
    "    \n",
    "    # transform the string to vector representation\n",
    "    key_tfidf = tfidf.transform([keywords]) \n",
    "    \n",
    "    # compute cosine similarity    \n",
    "    result = cosine_similarity(key_tfidf, tfidf_matrix)\n",
    "    \n",
    "    # sort top n similar movies   \n",
    "    similar_key_movies = sorted(list(enumerate(result[0])), reverse=True, key=lambda x: x[1])\n",
    "    \n",
    "    # extract names from dataframe and return movie names\n",
    "    recomm_mov = []\n",
    "    for i in similar_key_movies[1:n+1]:\n",
    "        recomm_mov.append(df.iloc[i[0]].title)\n",
    "        \n",
    "    return recomm_mov"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "0d75d6d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Insomnia', 'Man of Steel', 'Batman Begins', 'Interstellar']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recomm_keywords(\"Christopher Nolan\", 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "765c8bff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Harrison Montgomery', 'Scary Movie 4', 'Action Jackson', 'Dream House']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recomm_keywords(\"Daniel Craig\", 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2da0a1ec",
   "metadata": {},
   "source": [
    "Now, it’s time to switch to the last gear and transition from the notebook environment to a dynamic web application using Streamlit. To utilize the weights from the similarity matrix and the DataFrame effectively, we will save them as binary files using either joblib or pickle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ef8e23ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>corpus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>19995</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>7.2</td>\n",
       "      <td>In the 22nd century, a paraplegic Marine is di...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>285</td>\n",
       "      <td>Pirates of the Caribbean: At World's End</td>\n",
       "      <td>6.9</td>\n",
       "      <td>Captain Barbossa, long believed to be dead, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>206647</td>\n",
       "      <td>Spectre</td>\n",
       "      <td>6.3</td>\n",
       "      <td>A cryptic message from Bond’s past sends him o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49026</td>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>7.6</td>\n",
       "      <td>Following the death of District Attorney Harve...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49529</td>\n",
       "      <td>John Carter</td>\n",
       "      <td>6.1</td>\n",
       "      <td>John Carter is a war-weary, former military ca...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id                                     title  vote_average  \\\n",
       "0   19995                                    Avatar           7.2   \n",
       "1     285  Pirates of the Caribbean: At World's End           6.9   \n",
       "2  206647                                   Spectre           6.3   \n",
       "3   49026                     The Dark Knight Rises           7.6   \n",
       "4   49529                               John Carter           6.1   \n",
       "\n",
       "                                              corpus  \n",
       "0  In the 22nd century, a paraplegic Marine is di...  \n",
       "1  Captain Barbossa, long believed to be dead, ha...  \n",
       "2  A cryptic message from Bond’s past sends him o...  \n",
       "3  Following the death of District Attorney Harve...  \n",
       "4  John Carter is a war-weary, former military ca...  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "222da813",
   "metadata": {},
   "outputs": [],
   "source": [
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "c420d953",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['models/tfidf_mat.tf']"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(df, 'models/movie_db.df')\n",
    "joblib.dump(cos_mat, 'models/cos_mat.mt')\n",
    "joblib.dump(tfidf, 'models/vectorizer.tf')\n",
    "joblib.dump(tfidf_matrix, 'models/tfidf_mat.tf')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82b8650d",
   "metadata": {},
   "source": [
    "This way, we can easily load and utilize them within our web application. With these preparations, we bring this section to a fulfilling conclusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30eef8ec",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}