implicit_alternating_least_squares.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implicit Alternating Least Squares\n",
    "\n",
    "A recommendation model using an alternating least squares factorization approach for implicit datasets.\n",
    "\n",
    "[Iplicit ALS Documentation](https://implicit.readthedocs.io/en/latest/als.html)\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "* [Sample files](#sample-files)\n",
    "* [Step 1 - Prepare training data](#prepare-training-data)\n",
    " * [Download lastfm 360k dataset](#download)\n",
    " * [Prepare lastfm artist play training data](#prepare-data)\n",
    " * [Create training data file](#create-training-data-file)\n",
    " * [Upload training data file](#upload-training-data)\n",
    "* [Step 2 - Create a model](#create-model)\n",
    " * [Run a SageMaker training job](#run-training-job)\n",
    " * [Create a SageMaker model](#create-sagemaker-model)\n",
    "* [Step 3 - Get recommendations (inference)](#get-recommendations)\n",
    " * [Example users](#example-users)\n",
    " * [Create batch transform input file](#create-batch-input)\n",
    " * [Upload the batch transform input file to s3](#upload-batch-input)\n",
    " * [Run the Batch Transform Job](#run-transform)\n",
    " * [Download the batch results](#download-batch-results)\n",
    " * [Recommendations with scores](#recommendations)\n",
    " * [User history](#user-history)\n",
    "* [Step 4 - Optional Cleanup](#cleanup)\n",
    "\n",
    "## Sample files <a id=\"sample-files\"></a>\n",
    "\n",
    "These links are to example files on github.\n",
    "\n",
    "* [training input file](https://github.com/outpace/sagemaker-examples/blob/master/implicit-als/training/lastfm-360k-1mm-clean.csv)\n",
    "* [batch transform input file](https://github.com/outpace/sagemaker-examples/blob/master/implicit-als/batch_input/recommendation.requests)\n",
    "* [batch transform output file](https://github.com/outpace/sagemaker-examples/blob/master/implicit-als/recommendation.requests.out)\n",
    "\n",
    "## Step 1 - Prepare training data <a id=\"prepare-training-data\"></a>\n",
    "### Download lastfm 360k dataset <a id=\"download\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-11-08 17:33:11--  http://mtg.upf.edu/static/datasets/last.fm/lastfm-dataset-360K.tar.gz\n",
      "Resolving mtg.upf.edu (mtg.upf.edu)... 84.89.139.55\n",
      "Connecting to mtg.upf.edu (mtg.upf.edu)|84.89.139.55|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 569202935 (543M) [application/x-gzip]\n",
      "Saving to: ‘lastfm-dataset-360K.tar.gz’\n",
      "\n",
      "lastfm-dataset-360K 100%[===================>] 542.83M  1.53MB/s    in 5m 56s  \n",
      "\n",
      "2018-11-08 17:39:08 (1.52 MB/s) - ‘lastfm-dataset-360K.tar.gz’ saved [569202935/569202935]\n",
      "\n",
      "lastfm-dataset-360K/\n",
      "lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv\n",
      "lastfm-dataset-360K/README.txt\n",
      "lastfm-dataset-360K/mbox_sha1sum.py\n",
      "lastfm-dataset-360K/usersha1-profile.tsv\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\t3bd73256-3905-4f3a-97e2-8b341527f805\tbetty blowtorch\t2137\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\tf2fb0ff0-5679-42ec-a55c-15109ce6e320\tdie Ärzte\t1099\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\tb3ae82c2-e60b-4551-a76d-6620f1b456aa\tmelissa etheridge\t897\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\t3d6bbeb7-f90e-4d10-b440-e153c0d10b53\telvenking\t717\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\tbbd2ffd7-17f4-4506-8572-c1ea58c3f9a8\tjuliette & the licks\t706\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\t8bfac288-ccc5-448d-9573-c33ea2aa5c30\tred hot chili peppers\t691\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\t6531c8b1-76ea-4141-b270-eb1ac5b41375\tmagica\t545\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\t21f3573f-10cf-44b3-aeaa-26cccd8448b5\tthe black dahlia murder\t507\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\tc5db90c4-580d-4f33-b364-fbaa5a3a58b5\tthe murmurs\t424\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b\t0639533a-0402-40ba-b6e0-18b067198b73\tlunachicks\t403\n"
     ]
    }
   ],
   "source": [
    "!wget --no-clobber http://mtg.upf.edu/static/datasets/last.fm/lastfm-dataset-360K.tar.gz\n",
    "!tar -xvf lastfm-dataset-360K.tar.gz\n",
    "!head lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare lastfm artist play training data <a id=\"prepare-data\"></a>\n",
    "\n",
    "Import the tab separated lastfm file. Take only the first 1 million rows to save memory and processing time. Drop any rows with null values in `item_id`, `user_id`, `total_interactions`, `artist_name`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(987161, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>total_interactions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>3bd73256-3905-4f3a-97e2-8b341527f805</td>\n",
       "      <td>betty blowtorch</td>\n",
       "      <td>2137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>f2fb0ff0-5679-42ec-a55c-15109ce6e320</td>\n",
       "      <td>die Ärzte</td>\n",
       "      <td>1099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>b3ae82c2-e60b-4551-a76d-6620f1b456aa</td>\n",
       "      <td>melissa etheridge</td>\n",
       "      <td>897</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>3d6bbeb7-f90e-4d10-b440-e153c0d10b53</td>\n",
       "      <td>elvenking</td>\n",
       "      <td>717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>00000c289a1829a808ac09c00daf10bc3c4e223b</td>\n",
       "      <td>bbd2ffd7-17f4-4506-8572-c1ea58c3f9a8</td>\n",
       "      <td>juliette &amp; the licks</td>\n",
       "      <td>706</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    user_id  \\\n",
       "0  00000c289a1829a808ac09c00daf10bc3c4e223b   \n",
       "1  00000c289a1829a808ac09c00daf10bc3c4e223b   \n",
       "2  00000c289a1829a808ac09c00daf10bc3c4e223b   \n",
       "3  00000c289a1829a808ac09c00daf10bc3c4e223b   \n",
       "4  00000c289a1829a808ac09c00daf10bc3c4e223b   \n",
       "\n",
       "                                item_id           artist_name  \\\n",
       "0  3bd73256-3905-4f3a-97e2-8b341527f805       betty blowtorch   \n",
       "1  f2fb0ff0-5679-42ec-a55c-15109ce6e320             die Ärzte   \n",
       "2  b3ae82c2-e60b-4551-a76d-6620f1b456aa     melissa etheridge   \n",
       "3  3d6bbeb7-f90e-4d10-b440-e153c0d10b53             elvenking   \n",
       "4  bbd2ffd7-17f4-4506-8572-c1ea58c3f9a8  juliette & the licks   \n",
       "\n",
       "   total_interactions  \n",
       "0                2137  \n",
       "1                1099  \n",
       "2                 897  \n",
       "3                 717  \n",
       "4                 706  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv', \n",
    "                 sep=\"\\t\", \n",
    "                 header=None, \n",
    "                 names=[\"user_id\", \"item_id\", \"artist_name\", \"total_interactions\"], \n",
    "                 nrows=1000000)\n",
    "df = df.dropna(subset=['item_id', 'user_id', 'total_interactions', 'artist_name'])\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create training data file <a id=\"create-training-data-file\"></a>\n",
    "\n",
    "Create a csv file from the dataframe above. Do not include the index, but include headers `user_id`, `item_id`, and `total_interactions`. Show the head of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user_id,item_id,total_interactions\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,3bd73256-3905-4f3a-97e2-8b341527f805,2137\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,f2fb0ff0-5679-42ec-a55c-15109ce6e320,1099\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,b3ae82c2-e60b-4551-a76d-6620f1b456aa,897\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,3d6bbeb7-f90e-4d10-b440-e153c0d10b53,717\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,bbd2ffd7-17f4-4506-8572-c1ea58c3f9a8,706\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,8bfac288-ccc5-448d-9573-c33ea2aa5c30,691\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,6531c8b1-76ea-4141-b270-eb1ac5b41375,545\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,21f3573f-10cf-44b3-aeaa-26cccd8448b5,507\r\n",
      "00000c289a1829a808ac09c00daf10bc3c4e223b,c5db90c4-580d-4f33-b364-fbaa5a3a58b5,424\r\n"
     ]
    }
   ],
   "source": [
    "data_dir = 'implicit-als'\n",
    "train_data_dir = '{}/training'.format(data_dir)\n",
    "train_data_file = '{}/lastfm-360k-1mm-clean.csv'.format(train_data_dir)\n",
    "\n",
    "!mkdir -p {train_data_dir}\n",
    "df[[\"user_id\", \"item_id\", \"total_interactions\"]].to_csv(train_data_file, index=False)\n",
    "\n",
    "!head {train_data_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload training data to s3 <a id=\"upload-training-data\"></a>\n",
    "\n",
    "Choose a bucket, optionally customize the prefix, and upload the csv created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'uploaded training data file to s3://sagemaker-validation-us-east-2/implicit-als-test/training'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sagemaker as sage\n",
    "\n",
    "bucket = \"sagemaker-validation-us-east-2\"\n",
    "prefix = \"implicit-als-test\"\n",
    "\n",
    "sess = sage.Session()\n",
    "\n",
    "s3_train = sess.upload_data(train_data_dir, bucket, \"{}/training\".format(prefix))\n",
    "\"uploaded training data file to {}\".format(s3_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Create a model <a id=\"create-model\"></a>\n",
    "\n",
    "### Run a SageMaker training job <a id=\"run-training-job\"></a>\n",
    "\n",
    "This code will start a training job, wait for it to be done, and report its status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training job current status: InProgress\n",
      "Training job ended with status: Completed\n",
      "CPU times: user 167 ms, sys: 9.02 ms, total: 177 ms\n",
      "Wall time: 4min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "import boto3\n",
    "import time\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "role = get_execution_role()\n",
    "ecr_image = \"435525115971.dkr.ecr.us-east-2.amazonaws.com/sagemaker/implicit-als:9\"\n",
    "job_name_prefix = 'implicit-als-test'\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "job_name = job_name_prefix + timestamp\n",
    "create_training_params = \\\n",
    "{\n",
    "    \"AlgorithmSpecification\": {\n",
    "        \"TrainingImage\": ecr_image,\n",
    "        \"TrainingInputMode\": \"File\"\n",
    "    },\n",
    "    \"RoleArn\": role,\n",
    "    \"OutputDataConfig\": {\n",
    "        \"S3OutputPath\": 's3://{}/{}/output'.format(bucket, job_name_prefix)\n",
    "    },\n",
    "    \"ResourceConfig\": {\n",
    "        \"InstanceCount\": 1,\n",
    "        \"InstanceType\": \"ml.p3.2xlarge\",\n",
    "        \"VolumeSizeInGB\": 50\n",
    "    },\n",
    "    \"TrainingJobName\": job_name,\n",
    "    \"StoppingCondition\": {\n",
    "        \"MaxRuntimeInSeconds\": 360000\n",
    "    },\n",
    "    \"InputDataConfig\": [\n",
    "        {\n",
    "            \"ChannelName\": \"training\",\n",
    "            \"DataSource\": {\n",
    "                \"S3DataSource\": {\n",
    "                    \"S3DataType\": \"S3Prefix\",\n",
    "                    \"S3Uri\": s3_train,\n",
    "                    \"S3DataDistributionType\": \"FullyReplicated\"\n",
    "                }\n",
    "            },\n",
    "            \"ContentType\": \"application/csv\",\n",
    "            \"CompressionType\": \"None\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "sagemaker = boto3.client(service_name='sagemaker')\n",
    "sagemaker.create_training_job(**create_training_params)\n",
    "status = sagemaker.describe_training_job(TrainingJobName=job_name)['TrainingJobStatus']\n",
    "print('Training job current status: {}'.format(status))\n",
    "\n",
    "try:\n",
    "    sagemaker.get_waiter('training_job_completed_or_stopped').wait(TrainingJobName=job_name)\n",
    "    job_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "    status = job_info['TrainingJobStatus']\n",
    "    print(\"Training job ended with status: \" + status)\n",
    "except:\n",
    "    print('Training failed to start')\n",
    "    message = sagemaker.describe_training_job(TrainingJobName=job_name)['FailureReason']\n",
    "    print('Training failed with the following error: {}'.format(message))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a SageMaker model <a id=\"create-sagemaker-model\"></a>\n",
    "\n",
    "This will set up the model created during training within SageMaker to be used later for recommendations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ModelArn': 'arn:aws:sagemaker:us-east-2:435525115971:model/implicit-als-test-2018-11-08-17-43-37',\n",
       " 'ResponseMetadata': {'RequestId': 'a97c4ffb-fcb2-49b2-adac-719aeaa066da',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amzn-requestid': 'a97c4ffb-fcb2-49b2-adac-719aeaa066da',\n",
       "   'content-type': 'application/x-amz-json-1.1',\n",
       "   'content-length': '99',\n",
       "   'date': 'Thu, 08 Nov 2018 17:43:37 GMT'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "model_name=\"implicit-als-test\" + timestamp\n",
    "job_info = sagemaker.describe_training_job(TrainingJobName=job_name)\n",
    "model_data = job_info['ModelArtifacts']['S3ModelArtifacts']\n",
    "\n",
    "primary_container = {\n",
    "    'Image': ecr_image,\n",
    "    'ModelDataUrl': model_data,\n",
    "}\n",
    "\n",
    "create_model_response = sagemaker.create_model(\n",
    "    ModelName = model_name,\n",
    "    ExecutionRoleArn = role,\n",
    "    PrimaryContainer = primary_container)\n",
    "\n",
    "create_model_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Get recommendations (Inference) <a id=\"get-recommendations\"></a>\n",
    "\n",
    "### Example users <a id=\"example-users\"></a>\n",
    "\n",
    "Find some example users in order to predict their next rating/watch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>total_interactions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>207869</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>a7022764-95fb-46af-a7d6-90056746451a</td>\n",
       "      <td>uma thurman</td>\n",
       "      <td>651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207870</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>0743b15a-3c32-48c8-ad58-cb325350befa</td>\n",
       "      <td>blink-182</td>\n",
       "      <td>649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391892</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>99d7b49c-c18e-4a11-bf3e-b71710938df6</td>\n",
       "      <td>phoenix</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391893</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>bd4d397a-849a-48bf-be24-52eec87feeee</td>\n",
       "      <td>adriana calcanhotto</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         user_id  \\\n",
       "207869  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "207870  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "391892  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "391893  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "\n",
       "                                     item_id          artist_name  \\\n",
       "207869  a7022764-95fb-46af-a7d6-90056746451a          uma thurman   \n",
       "207870  0743b15a-3c32-48c8-ad58-cb325350befa            blink-182   \n",
       "391892  99d7b49c-c18e-4a11-bf3e-b71710938df6              phoenix   \n",
       "391893  bd4d397a-849a-48bf-be24-52eec87feeee  adriana calcanhotto   \n",
       "\n",
       "        total_interactions  \n",
       "207869                 651  \n",
       "207870                 649  \n",
       "391892                   3  \n",
       "391893                   2  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_users = df[df.user_id.isin([\"05c4bbb936abd2331e8f64037c95a61335d40e30\",\n",
    "                                    \"030ebbd1d8b360ce465a20e30a67a43da97f1b20\"])]\n",
    "example_users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create batch transform input file <a id=\"create-batch-input\"></a>\n",
    "\n",
    "Each row is a json object containing two keys:\n",
    "\n",
    "* `user_id`: the id of user\n",
    "* `top_n`: the number of top scoring recommendations to return\n",
    "\n",
    "The head of the batch input file is shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"user_id\": \"05c4bbb936abd2331e8f64037c95a61335d40e30\", \"top_n\": \"5\"}\r\n",
      "{\"user_id\": \"030ebbd1d8b360ce465a20e30a67a43da97f1b20\", \"top_n\": \"5\"}"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "batch_input_dir = '{}/batch_input'.format(data_dir)\n",
    "batch_input_file = batch_input_dir + '/recommendation.requests'\n",
    "\n",
    "!mkdir -p {batch_input_dir}\n",
    "\n",
    "with open(batch_input_file, 'w') as outfile:\n",
    "    json.dump({\"user_id\": \"05c4bbb936abd2331e8f64037c95a61335d40e30\", \"top_n\": \"5\"}, outfile)\n",
    "    outfile.write(\"\\n\")\n",
    "    json.dump({\"user_id\": \"030ebbd1d8b360ce465a20e30a67a43da97f1b20\", \"top_n\": \"5\"}, outfile)\n",
    "   \n",
    "!head {batch_input_file}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the batch transform input file to s3 <a id=\"upload-batch-input\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'uploaded training data file to s3://sagemaker-validation-us-east-2/implicit-als-test/batch_input'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_input = sess.upload_data(batch_input_dir, bucket, \"{}/batch_input\".format(prefix))\n",
    "\"uploaded training data file to {}\".format(batch_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the Batch Transform Job <a id=\"run-transform\"></a>\n",
    "\n",
    "This code will start a batch transform job, wait for it to be done, and report its status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created Transform job with name:  implicit-als-test-2018-11-08-17-43-38\n",
      "Transform job ended with status: Completed\n",
      "CPU times: user 92 ms, sys: 13.4 ms, total: 105 ms\n",
      "Wall time: 4min 1s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "timestamp = time.strftime('-%Y-%m-%d-%H-%M-%S', time.gmtime())\n",
    "batch_job_name = \"implicit-als-test\" + timestamp\n",
    "batch_output = 's3://{}/{}/output'.format(bucket, batch_job_name)\n",
    "request = \\\n",
    "{\n",
    "  \"TransformJobName\": batch_job_name,\n",
    "  \"ModelName\": model_name,\n",
    "  \"BatchStrategy\": \"SingleRecord\",\n",
    "  \"TransformInput\": {\n",
    "    \"DataSource\": {\n",
    "      \"S3DataSource\": {\n",
    "        \"S3DataType\": \"S3Prefix\",\n",
    "        \"S3Uri\": batch_input\n",
    "      }\n",
    "    },\n",
    "    \"ContentType\": \"application/json\",\n",
    "    \"CompressionType\": \"None\",\n",
    "    \"SplitType\": \"Line\"\n",
    "  },\n",
    "  \"TransformOutput\": {\n",
    "    \"S3OutputPath\": batch_output,\n",
    "    \"Accept\": \"text/csv\",\n",
    "    \"AssembleWith\": \"Line\"\n",
    "  },\n",
    "  \"TransformResources\": {\n",
    "    \"InstanceType\": \"ml.p3.2xlarge\",\n",
    "    \"InstanceCount\": 1\n",
    "  }\n",
    "}\n",
    "\n",
    "sagemaker.create_transform_job(**request)\n",
    "\n",
    "print(\"Created Transform job with name: \", batch_job_name)\n",
    "\n",
    "while(True):\n",
    "    job_info = sagemaker.describe_transform_job(TransformJobName=batch_job_name)\n",
    "    status = job_info['TransformJobStatus']\n",
    "    if status == 'Completed':\n",
    "        print(\"Transform job ended with status: \" + status)\n",
    "        break\n",
    "    if status == 'Failed':\n",
    "        message = job_info['FailureReason']\n",
    "        print('Transform failed with the following error: {}'.format(message))\n",
    "        raise Exception('Transform job failed') \n",
    "    time.sleep(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the batch results <a id=\"download-batch-results\"></a>\n",
    "\n",
    "Show the head of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "download: s3://sagemaker-validation-us-east-2/implicit-als-test-2018-11-08-17-43-38/output/recommendation.requests.out to implicit-als/recommendation.requests.out\n",
      "05c4bbb936abd2331e8f64037c95a61335d40e30,bd4d397a-849a-48bf-be24-52eec87feeee,0.024679340422153473\n",
      "05c4bbb936abd2331e8f64037c95a61335d40e30,99d7b49c-c18e-4a11-bf3e-b71710938df6,0.0211029015481472\n",
      "05c4bbb936abd2331e8f64037c95a61335d40e30,f81f19b9-c76e-43ac-8656-bb56071785fb,0.02010958269238472\n",
      "05c4bbb936abd2331e8f64037c95a61335d40e30,f07dbc2f-317b-470f-bad4-5f1b0eb6faf1,0.0188058540225029\n",
      "05c4bbb936abd2331e8f64037c95a61335d40e30,557ac15e-7800-44c3-bb62-d4ab34ac1e28,0.017942411825060844\n",
      "030ebbd1d8b360ce465a20e30a67a43da97f1b20,0743b15a-3c32-48c8-ad58-cb325350befa,0.7281188368797302\n",
      "030ebbd1d8b360ce465a20e30a67a43da97f1b20,a7022764-95fb-46af-a7d6-90056746451a,0.7025371193885803\n",
      "030ebbd1d8b360ce465a20e30a67a43da97f1b20,c2a44e93-3a2b-44aa-bd8b-7a71bb76e3b5,0.435314804315567\n",
      "030ebbd1d8b360ce465a20e30a67a43da97f1b20,f2eef649-a6d5-4114-afba-e50ab26254d2,0.43430081009864807\n",
      "030ebbd1d8b360ce465a20e30a67a43da97f1b20,67f66c07-6e61-4026-ade5-7e782fad3a5d,0.39594608545303345\n"
     ]
    }
   ],
   "source": [
    "!aws s3 cp {batch_output + '/recommendation.requests.out'} {data_dir}\n",
    "\n",
    "!head {data_dir}/recommendation.requests.out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommendations with scores <a id=\"recommendations\"></a>\n",
    "\n",
    "Import the recommendations from the batch output file downloaded above and join with artist names. These are the top 5 artist recommendations for our example users."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>score</th>\n",
       "      <th>artist_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>bd4d397a-849a-48bf-be24-52eec87feeee</td>\n",
       "      <td>0.024679</td>\n",
       "      <td>adriana calcanhotto</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>99d7b49c-c18e-4a11-bf3e-b71710938df6</td>\n",
       "      <td>0.021103</td>\n",
       "      <td>phoenix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>f81f19b9-c76e-43ac-8656-bb56071785fb</td>\n",
       "      <td>0.020110</td>\n",
       "      <td>marisa monte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>f07dbc2f-317b-470f-bad4-5f1b0eb6faf1</td>\n",
       "      <td>0.018806</td>\n",
       "      <td>caetano veloso</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>557ac15e-7800-44c3-bb62-d4ab34ac1e28</td>\n",
       "      <td>0.017942</td>\n",
       "      <td>los hermanos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>0743b15a-3c32-48c8-ad58-cb325350befa</td>\n",
       "      <td>0.728119</td>\n",
       "      <td>blink-182</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>a7022764-95fb-46af-a7d6-90056746451a</td>\n",
       "      <td>0.702537</td>\n",
       "      <td>uma thurman</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>c2a44e93-3a2b-44aa-bd8b-7a71bb76e3b5</td>\n",
       "      <td>0.435315</td>\n",
       "      <td>+44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>f2eef649-a6d5-4114-afba-e50ab26254d2</td>\n",
       "      <td>0.434301</td>\n",
       "      <td>sum 41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>67f66c07-6e61-4026-ade5-7e782fad3a5d</td>\n",
       "      <td>0.395946</td>\n",
       "      <td>foo fighters</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    user_id  \\\n",
       "0  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "1  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "2  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "3  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "4  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "5  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "6  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "7  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "8  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "9  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "\n",
       "                                item_id     score          artist_name  \n",
       "0  bd4d397a-849a-48bf-be24-52eec87feeee  0.024679  adriana calcanhotto  \n",
       "1  99d7b49c-c18e-4a11-bf3e-b71710938df6  0.021103              phoenix  \n",
       "2  f81f19b9-c76e-43ac-8656-bb56071785fb  0.020110         marisa monte  \n",
       "3  f07dbc2f-317b-470f-bad4-5f1b0eb6faf1  0.018806       caetano veloso  \n",
       "4  557ac15e-7800-44c3-bb62-d4ab34ac1e28  0.017942         los hermanos  \n",
       "5  0743b15a-3c32-48c8-ad58-cb325350befa  0.728119            blink-182  \n",
       "6  a7022764-95fb-46af-a7d6-90056746451a  0.702537          uma thurman  \n",
       "7  c2a44e93-3a2b-44aa-bd8b-7a71bb76e3b5  0.435315                  +44  \n",
       "8  f2eef649-a6d5-4114-afba-e50ab26254d2  0.434301               sum 41  \n",
       "9  67f66c07-6e61-4026-ade5-7e782fad3a5d  0.395946         foo fighters  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommendations_df = pd.read_csv('{}/recommendation.requests.out'.format(data_dir), \n",
    "                                 header=None, \n",
    "                                 names=[\"user_id\", \"item_id\", \"score\"])\n",
    "artist_names = df.groupby(['item_id']).agg(lambda x: x.iloc[0])[[\"artist_name\"]]\n",
    "recommendations_df = recommendations_df.join(artist_names, on='item_id')\n",
    "recommendations_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### User history <a id=\"user-history\"></a>\n",
    "\n",
    "Show the example users' history again for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>artist_name</th>\n",
       "      <th>total_interactions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>207869</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>a7022764-95fb-46af-a7d6-90056746451a</td>\n",
       "      <td>uma thurman</td>\n",
       "      <td>651</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207870</th>\n",
       "      <td>030ebbd1d8b360ce465a20e30a67a43da97f1b20</td>\n",
       "      <td>0743b15a-3c32-48c8-ad58-cb325350befa</td>\n",
       "      <td>blink-182</td>\n",
       "      <td>649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391892</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>99d7b49c-c18e-4a11-bf3e-b71710938df6</td>\n",
       "      <td>phoenix</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>391893</th>\n",
       "      <td>05c4bbb936abd2331e8f64037c95a61335d40e30</td>\n",
       "      <td>bd4d397a-849a-48bf-be24-52eec87feeee</td>\n",
       "      <td>adriana calcanhotto</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         user_id  \\\n",
       "207869  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "207870  030ebbd1d8b360ce465a20e30a67a43da97f1b20   \n",
       "391892  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "391893  05c4bbb936abd2331e8f64037c95a61335d40e30   \n",
       "\n",
       "                                     item_id          artist_name  \\\n",
       "207869  a7022764-95fb-46af-a7d6-90056746451a          uma thurman   \n",
       "207870  0743b15a-3c32-48c8-ad58-cb325350befa            blink-182   \n",
       "391892  99d7b49c-c18e-4a11-bf3e-b71710938df6              phoenix   \n",
       "391893  bd4d397a-849a-48bf-be24-52eec87feeee  adriana calcanhotto   \n",
       "\n",
       "        total_interactions  \n",
       "207869                 651  \n",
       "207870                 649  \n",
       "391892                   3  \n",
       "391893                   2  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Optional Clean up <a id=\"cleanup\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cleanup():\n",
    "    !rm lastfm-dataset-360K.tar.gz 2> /dev/null\n",
    "    !rm -fr implicit-als/ 2> /dev/null\n",
    "    !rm -fr lastfm-dataset-360K/ 2> /dev/null\n",
    "    sagemaker.delete_model(ModelName = model_name)\n",
    "    \n",
    "# optionally uncomment and run the code to clean everything up  \n",
    "\n",
    "#cleanup()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p36",
   "language": "python",
   "name": "conda_pytorch_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}