spotlight_explicit_factorization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spotlight Explicit Factorization\n",
    "\n",
    "An explicit feedback matrix factorization model. Uses a classic matrix factorization approach, with latent vectors used to represent both users and items. Their dot product gives the predicted score for a user-item pair.\n",
    "\n",
    "[Spotlight Documentation](https://maciejkula.github.io/spotlight/factorization/explicit.html)\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "* [Sample files](#sample-files)\n",
    "* [Step 1 - Prepare training data](#prepare-training-data)\n",
    " * [Download movielens 100k dataset](#download-movielens)\n",
    " * [Import ratings data](#import-ratings-data)\n",
    " * [Create training data file](#create-training-data-file)\n",
    " * [Upload training data file](#upload-training-data)\n",
    "* [Step 2 - Create a model](#create-model)\n",
    " * [Run a SageMaker training job](#run-training-job)\n",
    " * [Create a SageMaker model](#create-sagemaker-model)\n",
    "* [Step 3 - Get recommendations (inference)](#get-recommendations)\n",
    " * [Create batch transform input file](#create-batch-input)\n",
    " * [Upload the batch transform input file to s3](#upload-batch-input)\n",
    " * [Download the batch results](#download-batch-results)\n",
    " * [Import movie titles](#import-movie-titles)\n",
    " * [Recommendations with scores](#recommendations)\n",
    " * [User history](#user-history)\n",
    "* [Step 4 - Optional Cleanup](#cleanup)\n",
    "\n",
    "## Sample files <a id=\"sample-files\"></a>\n",
    "\n",
    "These links are to example files on github.\n",
    "\n",
    "* [training input file](https://github.com/outpace/sagemaker-examples/blob/master/train_data/ml-100k.csv)\n",
    "* [batch transform input file](https://github.com/outpace/sagemaker-examples/blob/master/spotlight-explicit-batch-input/recommendation.requests)\n",
    "* [batch transform output file](https://github.com/outpace/sagemaker-examples/blob/master/spotlight-explicit/recommendation.requests.out)\n",
    "\n",
    "## Step 1 - Prepare training data <a id=\"prepare-training-data\"></a>\n",
    "### Download movielens 100k dataset <a id=\"download-movielens\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-10-29 18:07:42--  http://files.grouplens.org/datasets/movielens/ml-100k.zip\n",
      "Resolving files.grouplens.org (files.grouplens.org)... 128.101.34.235\n",
      "Connecting to files.grouplens.org (files.grouplens.org)|128.101.34.235|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 4924029 (4.7M) [application/zip]\n",
      "Saving to: ‘ml-100k.zip’\n",
      "\n",
      "ml-100k.zip         100%[===================>]   4.70M  17.6MB/s    in 0.3s    \n",
      "\n",
      "2018-10-29 18:07:43 (17.6 MB/s) - ‘ml-100k.zip’ saved [4924029/4924029]\n",
      "\n",
      "Archive:  ml-100k.zip\n",
      "   creating: ml-100k/\n",
      "  inflating: ml-100k/allbut.pl       \n",
      "  inflating: ml-100k/mku.sh          \n",
      "  inflating: ml-100k/README          \n",
      "  inflating: ml-100k/u.data          \n",
      "  inflating: ml-100k/u.genre         \n",
      "  inflating: ml-100k/u.info          \n",
      "  inflating: ml-100k/u.item          \n",
      "  inflating: ml-100k/u.occupation    \n",
      "  inflating: ml-100k/u.user          \n",
      "  inflating: ml-100k/u1.base         \n",
      "  inflating: ml-100k/u1.test         \n",
      "  inflating: ml-100k/u2.base         \n",
      "  inflating: ml-100k/u2.test         \n",
      "  inflating: ml-100k/u3.base         \n",
      "  inflating: ml-100k/u3.test         \n",
      "  inflating: ml-100k/u4.base         \n",
      "  inflating: ml-100k/u4.test         \n",
      "  inflating: ml-100k/u5.base         \n",
      "  inflating: ml-100k/u5.test         \n",
      "  inflating: ml-100k/ua.base         \n",
      "  inflating: ml-100k/ua.test         \n",
      "  inflating: ml-100k/ub.base         \n",
      "  inflating: ml-100k/ub.test         \n"
     ]
    }
   ],
   "source": [
    "!wget --no-clobber http://files.grouplens.org/datasets/movielens/ml-100k.zip\n",
    "!unzip -o ml-100k.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import ratings data <a id=\"import-ratings-data\"></a>\n",
    "\n",
    "Drop the timestamp as it's not needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  item_id  rating\n",
       "0      196      242       3\n",
       "1      186      302       3\n",
       "2       22      377       1\n",
       "3      244       51       2\n",
       "4      166      346       1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "explicit_df = pd.read_csv('ml-100k/u.data', sep=\"\\t\", header=None, names=[\"user_id\", \"item_id\", \"rating\", \"timestamp\"]).drop([\"timestamp\"], axis=1)\n",
    "explicit_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create training data file <a id=\"create-training-data-file\"></a>\n",
    "\n",
    "Create a csv file from the dataframe above. Do not include the index, but include headers `user_id`, `item_id`, and `rating`. Show the head of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user_id,item_id,rating\r\n",
      "196,242,3\r\n",
      "186,302,3\r\n",
      "22,377,1\r\n",
      "244,51,2\r\n",
      "166,346,1\r\n",
      "298,474,4\r\n",
      "115,265,2\r\n",
      "253,465,5\r\n",
      "305,451,3\r\n"
     ]
    }
   ],
   "source": [
    "train_data_dir = 'spotlight_explicit_train_data'\n",
    "train_data_file = '{}/ml-100k.csv'.format(train_data_dir)\n",
    "\n",
    "!mkdir -p {train_data_dir}\n",
    "explicit_df.to_csv(train_data_file, index=False)\n",
    "\n",
    "!head {train_data_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload training data to s3 <a id=\"upload-training-data\"></a>\n",
    "\n",
    "Choose a bucket, optionally customize the prefix, and upload the csv created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'uploaded training data file to s3://sagemaker-validation-us-east-2/spotlight-explicit-factorization-test/training'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sagemaker as sage\n",
    "\n",
    "bucket = \"sagemaker-validation-us-east-2\"\n",
    "prefix = \"spotlight-explicit-factorization-test\"\n",
    "\n",
    "sess = sage.Session()\n",
    "\n",
    "s3_train = sess.upload_data(train_data_dir, bucket, \"{}/training\".format(prefix))\n",
    "\"uploaded training data file to {}\".format(s3_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Create a model <a id=\"create-model\"></a>\n",
    "\n",
    "### Run a SageMaker training job <a id=\"run-training-job\"></a>\n",
    "\n",
    "This code will start a training job, wait for it to be done, and report its status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training job current status: InProgress\n",
      "Training job ended with status: Completed\n",
      "CPU times: user 132 ms, sys: 24.6 ms, total: 156 ms\n",
      "Wall time: 4min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import boto3\n",
    "import time\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "ecr_image = \"435525115971.dkr.ecr.us-east-2.amazonaws.com/sagemaker/spotlight-explicit:9\"\n",
    "job_name_prefix = 'spotlight-expicit-factorization-test'\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "job_name = job_name_prefix + timestamp\n",
    "create_training_params = \\\n",
    "{\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": ecr_image,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"RoleArn\": role,\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": 's3://{}/{}/output'.format(bucket, job_name_prefix)\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,\n",
    "        \"InstanceType\": \"ml.p3.2xlarge\",\n",
    "        \"VolumeSizeInGB\": 50\n",
    "    },\n",
    "    \"TrainingJobName\": job_name,\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 360000\n",
    "    },\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"training\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": s3_train,\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"application/csv\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "sagemaker = boto3.client(service_name='sagemaker')\n",
    "sagemaker.create_training_job(**create_training_params)\n",
    "status = sagemaker.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']\n",
    "print('Training job current status: {}'.format(status))\n",
    "\n",
    "try:\n",
    "    sagemaker.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=job_name)\n",
    "    job_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "    status = job_info['TrainingJobStatus']\n",
    "    print(\"Training job ended with status: \" + status)\n",
    "except:\n",
    "    print('Training failed to start')\n",
    "    message = sagemaker.describe_training_job(TrainingJobName=job_name)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a SageMaker model <a id=\"create-sagemaker-model\"></a>\n",
    "\n",
    "This will set up the model created during training within SageMaker to be used later for recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ModelArn': 'arn:aws:sagemaker:us-east-2:435525115971:model/spotlight-explicit-factorization-test-2018-10-29-18-11-45',\n",
       " 'ResponseMetadata': {'RequestId': '37e677db-2b74-45c8-abe4-3085df6140b2',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': '37e677db-2b74-45c8-abe4-3085df6140b2',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '119',\n",
       "   'date': 'Mon, 29 Oct 2018 18:11:45 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "model_name=\"spotlight-explicit-factorization-test\" + timestamp\n",
    "job_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "model_data = job_info['ModelArtifacts']['S3ModelArtifacts']\n",
    "\n",
    "primary_container = {\n",
    "    'Image': ecr_image,\n",
    "    'ModelDataUrl': model_data,\n",
    "}\n",
    "\n",
    "create_model_response = sagemaker.create_model(\n",
    "    ModelName = model_name,\n",
    "    ExecutionRoleArn = role,\n",
    "    PrimaryContainer = primary_container)\n",
    "\n",
    "create_model_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Get recommendations (Inference) <a id=\"get-recommendations\"></a>\n",
    "\n",
    "### Create batch transform input file <a id=\"create-batch-input\"></a>\n",
    "\n",
    "Each row is a json object containing two keys:\n",
    "\n",
    "* `user_id`: the id of the user to get recommendations for\n",
    "* `top_n`: the number of top scoring recommendations to return\n",
    "\n",
    "The head of the batch input file is shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"user_id\": \"685\", \"top_n\": \"5\"}\r\n",
      "{\"user_id\": \"302\", \"top_n\": \"5\"}"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "batch_input_dir = 'spotlight-explicit-batch-input'\n",
    "batch_input_file = batch_input_dir + '/recommendation.requests'\n",
    "\n",
    "!mkdir -p {batch_input_dir}\n",
    "\n",
    "with open(batch_input_file, 'w') as outfile:\n",
    "    json.dump({\"user_id\": \"685\", \"top_n\": \"5\"}, outfile)\n",
    "    outfile.write(\"\\n\")\n",
    "    json.dump({\"user_id\": \"302\", \"top_n\": \"5\"}, outfile)\n",
    "   \n",
    "!head {batch_input_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the batch transform input file to s3 <a id=\"upload-batch-input\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'uploaded training data file to s3://sagemaker-validation-us-east-2/spotlight-explicit-factorization-test/batch_input'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_input = sess.upload_data(batch_input_dir, bucket, \"{}/batch_input\".format(prefix))\n",
    "\"uploaded training data file to {}\".format(batch_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the Batch Transform Job\n",
    "\n",
    "This code will start a batch transform job, wait for it to be done, and report its status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Transform job with name:  spotlight-explicit-factorization-test-2018-10-29-18-11-46\n",
      "Transform job ended with status: Completed\n",
      "CPU times: user 125 ms, sys: 1.69 ms, total: 127 ms\n",
      "Wall time: 5min 1s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "batch_job_name = \"spotlight-explicit-factorization-test\" + timestamp\n",
    "batch_output = 's3://{}/{}/output'.format(bucket, batch_job_name)\n",
    "request = \\\n",
    "{\n",
    "  \"TransformJobName\": batch_job_name,\n",
    "  \"ModelName\": model_name,\n",
    "  \"BatchStrategy\": \"SingleRecord\",\n",
    "  \"TransformInput\": {\n",
    "    \"DataSource\": {\n",
    "      \"S3DataSource\": {\n",
    "        \"S3DataType\": \"S3Prefix\",\n",
    "        \"S3Uri\": batch_input\n",
    "      }\n",
    "    },\n",
    "    \"ContentType\": \"application/json\",\n",
    "    \"CompressionType\": \"None\",\n",
    "    \"SplitType\": \"Line\"\n",
    "  },\n",
    "  \"TransformOutput\": {\n",
    "    \"S3OutputPath\": batch_output,\n",
    "    \"Accept\": \"text/csv\",\n",
    "    \"AssembleWith\": \"Line\"\n",
    "  },\n",
    "  \"TransformResources\": {\n",
    "    \"InstanceType\": \"ml.p3.2xlarge\",\n",
    "    \"InstanceCount\": 1\n",
    "  }\n",
    "}\n",
    "\n",
    "sagemaker.create_transform_job(**request)\n",
    "\n",
    "print(\"Created Transform job with name: \", batch_job_name)\n",
    "\n",
    "while(True):\n",
    "    job_info = sagemaker.describe_transform_job(TransformJobName=batch_job_name)\n",
    "    status = job_info['TransformJobStatus']\n",
    "    if status == 'Completed':\n",
    "        print(\"Transform job ended with status: \" + status)\n",
    "        break\n",
    "    if status == 'Failed':\n",
    "        message = job_info['FailureReason']\n",
    "        print('Transform failed with the following error: {}'.format(message))\n",
    "        raise Exception('Transform job failed') \n",
    "    time.sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the batch results <a id=\"download-batch-results\"></a>\n",
    "\n",
    "Show the head of the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-validation-us-east-2/spotlight-explicit-factorization-test-2018-10-29-18-11-46/output/recommendation.requests.out to spotlight-explicit/recommendation.requests.out\n",
      "685,1643,5.474481582641602\n",
      "685,48,5.30016565322876\n",
      "685,956,5.123522758483887\n",
      "685,276,5.095803260803223\n",
      "685,507,4.989534854888916\n",
      "302,752,4.704867362976074\n",
      "302,344,4.447729587554932\n",
      "302,288,4.417259216308594\n",
      "302,271,4.409123420715332\n",
      "302,27,4.326932907104492\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p spotlight-explicit/\n",
    "\n",
    "!aws s3 cp {batch_output + '/recommendation.requests.out'} spotlight-explicit/\n",
    "\n",
    "!head spotlight-explicit/recommendation.requests.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import movie titles <a id=\"import-movie-titles\"></a>\n",
    "\n",
    "Get movie titles in `u.item` from the movielens files downloaded earlier and join with ratings data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>movie_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3</td>\n",
       "      <td>Kolya (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>L.A. Confidential (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1</td>\n",
       "      <td>Heavyweights (1994)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>Legends of the Fall (1994)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1</td>\n",
       "      <td>Jackie Brown (1997)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  item_id  rating                 movie_title\n",
       "0      196      242       3                Kolya (1996)\n",
       "1      186      302       3    L.A. Confidential (1997)\n",
       "2       22      377       1         Heavyweights (1994)\n",
       "3      244       51       2  Legends of the Fall (1994)\n",
       "4      166      346       1         Jackie Brown (1997)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_df = pd.read_csv('ml-100k/u.item', sep=\"|\", header=None, encoding = \"ISO-8859-1\").set_index([0]).iloc[:,0:1]\n",
    "explicit_df = explicit_df.join(titles_df, on='item_id').rename(index=str, columns={\"user_id\":\"user_id\",1:\"movie_title\"})\n",
    "explicit_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommendations with scores <a id=\"recommendations\"></a>\n",
    "\n",
    "Import the recommendations from the batch output file downloaded above and join with titles dataframe. These are the top 5 movie recommendations for users 685 and 302."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>score</th>\n",
       "      <th>movie_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>685</td>\n",
       "      <td>333</td>\n",
       "      <td>14.640368</td>\n",
       "      <td>Game, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>685</td>\n",
       "      <td>272</td>\n",
       "      <td>13.269605</td>\n",
       "      <td>Good Will Hunting (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>685</td>\n",
       "      <td>268</td>\n",
       "      <td>13.025604</td>\n",
       "      <td>Chasing Amy (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>685</td>\n",
       "      <td>347</td>\n",
       "      <td>12.637764</td>\n",
       "      <td>Wag the Dog (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>685</td>\n",
       "      <td>315</td>\n",
       "      <td>12.535725</td>\n",
       "      <td>Apt Pupil (1998)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>302</td>\n",
       "      <td>748</td>\n",
       "      <td>12.058822</td>\n",
       "      <td>Saint, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>302</td>\n",
       "      <td>333</td>\n",
       "      <td>11.834901</td>\n",
       "      <td>Game, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>302</td>\n",
       "      <td>323</td>\n",
       "      <td>11.719286</td>\n",
       "      <td>Dante's Peak (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>302</td>\n",
       "      <td>258</td>\n",
       "      <td>10.959321</td>\n",
       "      <td>Contact (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>302</td>\n",
       "      <td>313</td>\n",
       "      <td>10.367278</td>\n",
       "      <td>Titanic (1997)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id  item_id      score               movie_title\n",
       "0      685      333  14.640368          Game, The (1997)\n",
       "1      685      272  13.269605  Good Will Hunting (1997)\n",
       "2      685      268  13.025604        Chasing Amy (1997)\n",
       "3      685      347  12.637764        Wag the Dog (1997)\n",
       "4      685      315  12.535725          Apt Pupil (1998)\n",
       "5      302      748  12.058822         Saint, The (1997)\n",
       "6      302      333  11.834901          Game, The (1997)\n",
       "7      302      323  11.719286       Dante's Peak (1997)\n",
       "8      302      258  10.959321            Contact (1997)\n",
       "9      302      313  10.367278            Titanic (1997)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommendations_df = pd.read_csv('recommendation.requests.out', header=None, names=[\"user_id\", \"item_id\", \"score\"])\n",
    "recommendations_df = recommendations_df.join(titles_df, on='item_id').rename(index=str, columns={\"user_id\":\"user_id\",1:\"movie_title\"})\n",
    "recommendations_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User history <a id=\"user-history\"></a>\n",
    "\n",
    "For reference, here are the movies users 685 and 302 watched/rated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>rating</th>\n",
       "      <th>movie_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>44226</th>\n",
       "      <td>685</td>\n",
       "      <td>340</td>\n",
       "      <td>2</td>\n",
       "      <td>Boogie Nights (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63372</th>\n",
       "      <td>685</td>\n",
       "      <td>299</td>\n",
       "      <td>2</td>\n",
       "      <td>Hoodlum (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44177</th>\n",
       "      <td>685</td>\n",
       "      <td>319</td>\n",
       "      <td>2</td>\n",
       "      <td>Everyone Says I Love You (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47608</th>\n",
       "      <td>685</td>\n",
       "      <td>334</td>\n",
       "      <td>1</td>\n",
       "      <td>U Turn (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53414</th>\n",
       "      <td>685</td>\n",
       "      <td>325</td>\n",
       "      <td>3</td>\n",
       "      <td>Crash (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53471</th>\n",
       "      <td>685</td>\n",
       "      <td>327</td>\n",
       "      <td>2</td>\n",
       "      <td>Cop Land (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55210</th>\n",
       "      <td>685</td>\n",
       "      <td>873</td>\n",
       "      <td>2</td>\n",
       "      <td>Picture Perfect (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60486</th>\n",
       "      <td>685</td>\n",
       "      <td>337</td>\n",
       "      <td>2</td>\n",
       "      <td>House of Yes, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70388</th>\n",
       "      <td>685</td>\n",
       "      <td>324</td>\n",
       "      <td>3</td>\n",
       "      <td>Lost Highway (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43618</th>\n",
       "      <td>685</td>\n",
       "      <td>302</td>\n",
       "      <td>3</td>\n",
       "      <td>L.A. Confidential (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83124</th>\n",
       "      <td>685</td>\n",
       "      <td>872</td>\n",
       "      <td>2</td>\n",
       "      <td>Love Jones (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83388</th>\n",
       "      <td>685</td>\n",
       "      <td>333</td>\n",
       "      <td>1</td>\n",
       "      <td>Game, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89437</th>\n",
       "      <td>685</td>\n",
       "      <td>882</td>\n",
       "      <td>3</td>\n",
       "      <td>Washington Square (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98137</th>\n",
       "      <td>685</td>\n",
       "      <td>289</td>\n",
       "      <td>2</td>\n",
       "      <td>Evita (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98163</th>\n",
       "      <td>685</td>\n",
       "      <td>875</td>\n",
       "      <td>3</td>\n",
       "      <td>She's So Lovely (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98172</th>\n",
       "      <td>685</td>\n",
       "      <td>886</td>\n",
       "      <td>1</td>\n",
       "      <td>Life Less Ordinary, A (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43697</th>\n",
       "      <td>685</td>\n",
       "      <td>991</td>\n",
       "      <td>1</td>\n",
       "      <td>Keys to Tulsa (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98438</th>\n",
       "      <td>685</td>\n",
       "      <td>288</td>\n",
       "      <td>2</td>\n",
       "      <td>Scream (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43357</th>\n",
       "      <td>685</td>\n",
       "      <td>286</td>\n",
       "      <td>1</td>\n",
       "      <td>English Patient, The (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42723</th>\n",
       "      <td>685</td>\n",
       "      <td>269</td>\n",
       "      <td>3</td>\n",
       "      <td>Full Monty, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10611</th>\n",
       "      <td>302</td>\n",
       "      <td>270</td>\n",
       "      <td>2</td>\n",
       "      <td>Gattaca (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2593</th>\n",
       "      <td>302</td>\n",
       "      <td>323</td>\n",
       "      <td>2</td>\n",
       "      <td>Dante's Peak (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2858</th>\n",
       "      <td>302</td>\n",
       "      <td>245</td>\n",
       "      <td>2</td>\n",
       "      <td>Devil's Own, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3967</th>\n",
       "      <td>302</td>\n",
       "      <td>309</td>\n",
       "      <td>2</td>\n",
       "      <td>Deceiver (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4826</th>\n",
       "      <td>302</td>\n",
       "      <td>328</td>\n",
       "      <td>3</td>\n",
       "      <td>Conspiracy Theory (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87380</th>\n",
       "      <td>302</td>\n",
       "      <td>266</td>\n",
       "      <td>2</td>\n",
       "      <td>Kull the Conqueror (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84234</th>\n",
       "      <td>302</td>\n",
       "      <td>333</td>\n",
       "      <td>3</td>\n",
       "      <td>Game, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9848</th>\n",
       "      <td>302</td>\n",
       "      <td>307</td>\n",
       "      <td>4</td>\n",
       "      <td>Devil's Advocate, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10387</th>\n",
       "      <td>302</td>\n",
       "      <td>879</td>\n",
       "      <td>2</td>\n",
       "      <td>Peacemaker, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81824</th>\n",
       "      <td>302</td>\n",
       "      <td>271</td>\n",
       "      <td>4</td>\n",
       "      <td>Starship Troopers (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14758</th>\n",
       "      <td>302</td>\n",
       "      <td>258</td>\n",
       "      <td>3</td>\n",
       "      <td>Contact (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43238</th>\n",
       "      <td>302</td>\n",
       "      <td>748</td>\n",
       "      <td>1</td>\n",
       "      <td>Saint, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15594</th>\n",
       "      <td>302</td>\n",
       "      <td>299</td>\n",
       "      <td>2</td>\n",
       "      <td>Hoodlum (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58767</th>\n",
       "      <td>302</td>\n",
       "      <td>289</td>\n",
       "      <td>3</td>\n",
       "      <td>Evita (1996)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55280</th>\n",
       "      <td>302</td>\n",
       "      <td>358</td>\n",
       "      <td>3</td>\n",
       "      <td>Spawn (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32327</th>\n",
       "      <td>302</td>\n",
       "      <td>301</td>\n",
       "      <td>4</td>\n",
       "      <td>In &amp; Out (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32747</th>\n",
       "      <td>302</td>\n",
       "      <td>680</td>\n",
       "      <td>2</td>\n",
       "      <td>Kull the Conqueror (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33530</th>\n",
       "      <td>302</td>\n",
       "      <td>988</td>\n",
       "      <td>2</td>\n",
       "      <td>Beautician and the Beast, The (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50283</th>\n",
       "      <td>302</td>\n",
       "      <td>294</td>\n",
       "      <td>1</td>\n",
       "      <td>Liar Liar (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>302</td>\n",
       "      <td>303</td>\n",
       "      <td>2</td>\n",
       "      <td>Ulee's Gold (1997)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299</th>\n",
       "      <td>302</td>\n",
       "      <td>322</td>\n",
       "      <td>2</td>\n",
       "      <td>Murder at 1600 (1997)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user_id  item_id  rating                           movie_title\n",
       "44226      685      340       2                  Boogie Nights (1997)\n",
       "63372      685      299       2                        Hoodlum (1997)\n",
       "44177      685      319       2       Everyone Says I Love You (1996)\n",
       "47608      685      334       1                         U Turn (1997)\n",
       "53414      685      325       3                          Crash (1996)\n",
       "53471      685      327       2                       Cop Land (1997)\n",
       "55210      685      873       2                Picture Perfect (1997)\n",
       "60486      685      337       2              House of Yes, The (1997)\n",
       "70388      685      324       3                   Lost Highway (1997)\n",
       "43618      685      302       3              L.A. Confidential (1997)\n",
       "83124      685      872       2                     Love Jones (1997)\n",
       "83388      685      333       1                      Game, The (1997)\n",
       "89437      685      882       3              Washington Square (1997)\n",
       "98137      685      289       2                          Evita (1996)\n",
       "98163      685      875       3                She's So Lovely (1997)\n",
       "98172      685      886       1          Life Less Ordinary, A (1997)\n",
       "43697      685      991       1                  Keys to Tulsa (1997)\n",
       "98438      685      288       2                         Scream (1996)\n",
       "43357      685      286       1           English Patient, The (1996)\n",
       "42723      685      269       3                Full Monty, The (1997)\n",
       "10611      302      270       2                        Gattaca (1997)\n",
       "2593       302      323       2                   Dante's Peak (1997)\n",
       "2858       302      245       2               Devil's Own, The (1997)\n",
       "3967       302      309       2                       Deceiver (1997)\n",
       "4826       302      328       3              Conspiracy Theory (1997)\n",
       "87380      302      266       2             Kull the Conqueror (1997)\n",
       "84234      302      333       3                      Game, The (1997)\n",
       "9848       302      307       4          Devil's Advocate, The (1997)\n",
       "10387      302      879       2                Peacemaker, The (1997)\n",
       "81824      302      271       4              Starship Troopers (1997)\n",
       "14758      302      258       3                        Contact (1997)\n",
       "43238      302      748       1                     Saint, The (1997)\n",
       "15594      302      299       2                        Hoodlum (1997)\n",
       "58767      302      289       3                          Evita (1996)\n",
       "55280      302      358       3                          Spawn (1997)\n",
       "32327      302      301       4                       In & Out (1997)\n",
       "32747      302      680       2             Kull the Conqueror (1997)\n",
       "33530      302      988       2  Beautician and the Beast, The (1997)\n",
       "50283      302      294       1                      Liar Liar (1997)\n",
       "318        302      303       2                    Ulee's Gold (1997)\n",
       "299        302      322       2                 Murder at 1600 (1997)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explicit_df[explicit_df.user_id.isin([685,302])].sort_values(by=['user_id'], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Optional Clean up <a id=\"cleanup\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# optionally uncomment and run the code to clean everything up\n",
    "\n",
    "#!rm ml-100k.zip 2> /dev/null\n",
    "#!rm spotlight-explicit/recommendation.requests.out 2> /dev/null\n",
    "#!rm -fr ml-100k/ 2> /dev/null\n",
    "#!rm -fr {train_data_dir} 2> /dev/null\n",
    "#!rm -fr {batch_input_dir} 2> /dev/null\n",
    "#sagemaker.delete_model(ModelName= model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}