collaborative-filtering-model-based.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Based Collaborative Filtering Recommender\n",
    "\n",
    "The goal of the **recommender system** is to predict user preference for a set of items based on the past experience. Two the most popular approaches are Content-Based and Collaborative Filtering.\n",
    "\n",
    "**Collaborative filtering** is a technique used by websites like Amazon, YouTube, and Netflix. It filters out items that a user might like on the basis of reactions of similar users. There are two categories of collaborative filtering algorithms: memory based and model based.\n",
    "\n",
    "**Model based approach** involves building machine learning algorithms to predict user's ratings. They involve dimensionality reduction methods that reduce high dimensional matrix containing abundant number of missing values with a much smaller matrix in lower-dimensional space.\n",
    "\n",
    "The goal of this exercise is to compare SVD and NMF algorithms, try different configurations of parameters and explore obtained results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from surprise import Dataset, Reader\n",
    "from surprise import SVD, NMF\n",
    "from surprise.model_selection import cross_validate, train_test_split, GridSearchCV\n",
    "\n",
    "import functions as f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This analysis will focus on book recommendations based on [Book-Crossing dataset](http://www2.informatik.uni-freiburg.de/~cziegler/BX/). To reduce the dimensionality of the dataset and avoid running into memory error it will focus on users with at least 3 ratings and top 10% most frequently rated books. It consists of 176,594 records.\n",
    "\n",
    "The recommender systems will be built using [surprise package](https://surprise.readthedocs.io/en/stable/getting_started.html) (Matrix Factorization - based models)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/ratings_top.csv')\n",
    "\n",
    "reader = Reader(rating_scale=(1, 10))\n",
    "data = Dataset.load_from_df(df[['user_id', 'isbn', 'book_rating']], reader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of ratings: 176594\n",
      "Number of books: 16766\n",
      "Number of users: 20149\n"
     ]
    }
   ],
   "source": [
    "print('Number of ratings: %d\\nNumber of books: %d\\nNumber of users: %d' % (len(df), len(df['isbn'].unique()), len(df['user_id'].unique())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVD and NMF models comparison\n",
    "\n",
    "Singular Value Decomposition (SVD) and Non-negative Matrix Factorization (NMF) are matrix factorization techniques used for dimensionality reduction. Surprise package provides implementation of those algorithms.\n",
    "\n",
    "It's clear that for the given dataset much better results can be obtained with SVD approach - both in terms of accuracy and training / testing time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "test_rmse     1.606926\n",
       "test_mae      1.242338\n",
       "fit_time     18.130412\n",
       "test_time     1.120190\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_svd = SVD()\n",
    "cv_results_svd = cross_validate(model_svd, data, cv=3)\n",
    "pd.DataFrame(cv_results_svd).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "test_rmse     2.640803\n",
       "test_mae      2.255504\n",
       "fit_time     22.795353\n",
       "test_time     1.005285\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_nmf = NMF()\n",
    "cv_results_nmf = cross_validate(model_nmf, data, cv=3)\n",
    "pd.DataFrame(cv_results_nmf).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimisation of SVD algorithm\n",
    "\n",
    "Grid Search Cross Validation computes accuracy metrics for an algorithm on various combinations of parameters, over a cross-validation procedure. It's useful for finding the best configuration of parameters.\n",
    "\n",
    "It is used to find the best setting of parameters:\n",
    "* n_factors - the number of factors\n",
    "* n_epochs - the number of iteration of the SGD procedure\n",
    "* lr_all - the learning rate for all parameters\n",
    "* reg_all - the regularization term for all parameters\n",
    "\n",
    "As a result, regarding the majority of parameters, the default setting is the most optimal one. The improvement obtained with Grid Search is very small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.5981785240945765\n",
      "{'n_factors': 80, 'n_epochs': 20, 'lr_all': 0.005, 'reg_all': 0.2}\n"
     ]
    }
   ],
   "source": [
    "param_grid = {'n_factors': [80,100,120],\n",
    "              'n_epochs': [5, 10, 20],\n",
    "              'lr_all': [0.002, 0.005],\n",
    "              'reg_all': [0.2, 0.4, 0.6]}\n",
    "\n",
    "gs = GridSearchCV(SVD, param_grid, measures=['rmse', 'mae'], cv=3)\n",
    "gs.fit(data)\n",
    "\n",
    "print(gs.best_score['rmse'])\n",
    "print(gs.best_params['rmse'])\n",
    "\n",
    "#1.5981785240945765\n",
    "#{'n_factors': 80, 'n_epochs': 20, 'lr_all': 0.005, 'reg_all': 0.2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis of Collaborative Filtering model results\n",
    "\n",
    "In this part, let's examine in detail the results obtained by the SVD model that provided the best RMSE score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainset, testset = train_test_split(data, test_size=0.2)\n",
    "\n",
    "model = SVD(n_factors=80, n_epochs=20, lr_all=0.005, reg_all=0.2)\n",
    "model.fit(trainset)\n",
    "predictions = model.test(testset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>isbn</th>\n",
       "      <th>actual_rating</th>\n",
       "      <th>pred_rating</th>\n",
       "      <th>impossible</th>\n",
       "      <th>pred_rating_round</th>\n",
       "      <th>abs_err</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6118</th>\n",
       "      <td>242999</td>\n",
       "      <td>0345374568</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.246858</td>\n",
       "      <td>False</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.246858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17185</th>\n",
       "      <td>67840</td>\n",
       "      <td>155166951X</td>\n",
       "      <td>10.0</td>\n",
       "      <td>8.513183</td>\n",
       "      <td>False</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.486817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21313</th>\n",
       "      <td>78553</td>\n",
       "      <td>0451404327</td>\n",
       "      <td>10.0</td>\n",
       "      <td>9.083398</td>\n",
       "      <td>False</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.916602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23423</th>\n",
       "      <td>107784</td>\n",
       "      <td>0373031467</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.890978</td>\n",
       "      <td>False</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.890978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9899</th>\n",
       "      <td>95250</td>\n",
       "      <td>0375725601</td>\n",
       "      <td>9.0</td>\n",
       "      <td>8.035049</td>\n",
       "      <td>False</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.964951</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       user_id        isbn  actual_rating  pred_rating  impossible  \\\n",
       "6118    242999  0345374568            5.0     7.246858       False   \n",
       "17185    67840  155166951X           10.0     8.513183       False   \n",
       "21313    78553  0451404327           10.0     9.083398       False   \n",
       "23423   107784  0373031467            5.0     5.890978       False   \n",
       "9899     95250  0375725601            9.0     8.035049       False   \n",
       "\n",
       "       pred_rating_round   abs_err  \n",
       "6118                 7.0  2.246858  \n",
       "17185                9.0  1.486817  \n",
       "21313                9.0  0.916602  \n",
       "23423                6.0  0.890978  \n",
       "9899                 8.0  0.964951  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pred = pd.DataFrame(predictions, columns=['user_id', 'isbn', 'actual_rating', 'pred_rating', 'details'])\n",
    "\n",
    "df_pred['impossible'] = df_pred['details'].apply(lambda x: x['was_impossible'])\n",
    "df_pred['pred_rating_round'] = df_pred['pred_rating'].round()\n",
    "df_pred['abs_err'] = abs(df_pred['pred_rating'] - df_pred['actual_rating'])\n",
    "df_pred.drop(['details'], axis=1, inplace=True)\n",
    "\n",
    "df_pred.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution of actual and predicted ratings in the test set\n",
    "\n",
    "According to the distribution of actual ratings of books in the test set, the biggest part of users give positive scores - between 7 and 10. The mode equals 8 but count of ratings 7, 9, 10 is also noticeable. The distribution of predicted ratings in the test set is visibly different. One more time, 8 is a mode but scores 7, 9 and 10 are clearly less frequent.\n",
    "\n",
    "It shows that the recommender system is not perfect and it cannot reflect the real distribution of book ratings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAEXCAYAAAB4Yg/uAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZglZX33//dHFpFNQEaCAwgaJEGSIM4jJG4kKCIxookaeFwQSYgJJBp9fhGyPBCVBPOYGE0Ug4KAUUDFZWJQRBSJRpYBkUU0jIAyMMLgsLmh4Pf3R90NNT2ne7qnt9PT79d1navr3LV9q+qcuvt76q67UlVIkiRJkjqPmOsAJEmSJGmYmCRJkiRJUo9JkiRJkiT1mCRJkiRJUo9JkiRJkiT1mCRJkiRJUo9J0jRK8t4kfzNNy9olyQ+SbNTeX5TkD6Zj2W15n0ly+HQtbxLrfWuSO5N8b7bXPVFJXp3kyzO8juuS7D+T65iMJDskuTjJfUn+ccD405O8dQbWe0KSf5/iMp6Z5FvTGNOMbKs0k6x/JrTeOa1/klSSX2zD03a81rFO67O1x1ufaUJMkiYoyc1Jfty+dHcn+e8kr03y0D6sqtdW1VsmuKznjDdNVX23qrasqgenIfa1vrhV9fyqOmOqy55kHDsDbwT2rKpfmMbl7p9kxXQtb7oNOklV1ZOr6qI5CmmQo4A7ga2r6o1zHcxkVNV/VdUe6zPvTP8DMV3/XM7kZ3wi5yPNLeufqZup+md9TeJ4TWuCOlXWZzPL+mzKy57W+swkaXJ+p6q2Ah4PnAS8CTh1uleSZOPpXuaQeDzw/aq6Y64DmS4b0LF6PPCN8unS0rCy/pmaaa1/Rq6ybUg2oGNvfabpUVW+JvACbgaeM6rsacDPgb3a+9OBt7bh7YFPA3cDq4H/oktKP9jm+THwA+AvgF2BAo4Evgtc3CvbuC3vIuDvgcuAe4BPAdu1cfsDKwbFCxwE/BT4WVvf13vL+4M2/Ajgr4HvAHcAZwKPbuNG4ji8xXYn8Ffj7KdHt/lXteX9dVv+c9o2/7zFcfqAebdt+2wVcFcb3qk3fjvgA8BtbfwngS1GLfcHwOP6x2LQPgKOBb4N3Ad8A3hxb9yrgS+PsX1rHatW/lHge+3YXAw8uZUf1fb9T1ts/zH68wScAHyk7bf7gOuAJb117gN8rY37KHAO6/icjRH7bwCXtxgvB36j97ntx/icAfOeDrwXuKDF8SXg8etadhv3OGBpi2858Ie9cScA/96GNwHOAs4FNqX7fi0D7gVuB/5pjO0afWxvBv4PcHWL5xxgswHz/TLwE+DBtt1397b13cB/tm29FHhib75favthNfAt4GVjxHViW/ZP2vL/dV3zAwfTfR7vA25t2zHwMz5gfWvN2xv3AuCq9jn5b+BXW/la56O5Ptf6GvhZunn09xLrn0H7aSr1z/7ACuAv23puBl7eG386cDJwHvDDtsxHAm9vsd1Od458VG+e/w9YSVdnvaZtyy+OPl7t/SF039F76eqmg1i/c8hj6M6397bj9Rasz0bPezrWZyPzWZ+Nd+6d7pP5hvpiQCXVyr8L/HHvwzjyZf/79iXcpL2eCWTQsnj4RHVm+wA9isGV1K3AXm2ac3tfxjW+VKPXQe+L2xt/EQ9XUq+h+7I/AdgS+DjwwVGxva/F9WvA/cAvj7GfzqSrQLdq8/4PcORYcY6a9zHA7wGbt/k/CnyyN/4/6U4Q27Z9+uxxtv+hYzFoGuCldCe7RwC/T1fp7djGvZp1VyoPHavePtyKrtL8Z+CqsWIZ4/j8hO6ksBHdZ+eSNm5Tusr+dW2bf5fu5L/Oz9mo9W1Hl1i+EtgYOKy9f8xYMQ7Yn/cBz2rb+M6RfTSBZX8JeA+wGbA33T8wB/Q/m3Sfrf9s69mojfsq8Mo2vCWw3xixjT62N9P9c/C4Ftv1wGvHmHetY91iWE1XqW0MfAg4u43bArgFOKKN24fuH6onj7H8i2jfs4nMT/cP1TPb8LbAPhP57qxj3n3o/vncl+7zdXjbR48c79zma3heYx0jrH9G74+p1D/7Aw8A/0R3jns2Xb2wR2//3gM8na7e2IzuXL+U7jyzFfAfwN+36Q+i+2d4ZJ99mDGSJLpzzT3Ac9uyFwO/NHpftffrOoecTZekbNHWfSvWZ6PnPx3rM7A+W+fL5nZTdxvdB3e0nwE70v068bPq2pnWOpZ1QlX9sKp+PMb4D1bVtVX1Q+BvgJdN0yX/l9P9qnFjVf0AOA44dNSl97+tqh9X1deBr9NVVmtosfw+cFxV3VdVNwP/SHeyWaeq+n5VnVtVP6qq++h+uXh2W/aOwPPpTg53tX36pfXd4Kr6aFXdVlU/r6pzgBvoTiITtcaxqqrT2jbfT3ei/LUkj57E8r5cVedVdw/AB3l4/+5Hd/J5V9vmj9OdMEdM9HP228ANVfXBqnqgqs4Cvgn8ziRi/M+qurht418Bv97a+Y+57Db+GcCbquonVXUV8H7W/ExsDXyW7tfTI+rh+yB+Bvxiku2r6gdVdckkYn1XO76r6f5x2XsS8wJ8vKouq6oH6CqVkflfANxcVR9o23ol3T+ML5ngctc1/8+APZNs3T7nV04i5rHm/UPg36rq0qp6sLp7Qe6n+2xpfrP+aaZa//T8TVXd3+qX/wRe1hv3qar6SlX9nO479IfAn1fV6lZn/R1waJv2ZcAHevvshHHWeSRwWlVd0OqkW6vqm2NMO+Y5pO2D3wP+bzuW1wITuffL+sz6zPpsAJOkqVtMl6WP9v/ofh37XJIbkxw7gWXdMonx36H7pWX7CUU5vse15fWXvTGwQ6+s3xvQj+h+CRltex7+pai/rMUTCSLJ5kn+Lcl3ktxLd5l/m3bi3xlYXVV3TWRZE1jXq5Jc1W6CvpvuF7fJ7MuHjkWSjZKclOTbLe6b26jJLG/0/t2s/ZPwOODWURVF/3Mw0c/Z6GMMkzg2o9fb/plZ3ZY73rIfR3fc7htnvfsBvwqcNGo7jwSeBHwzyeVJXjCJWCfyeV2f+R8P7DvyuWmfnZcDE70RfF3z/x7dL7DfSfKlJL8+iZjHmvfxwBtHrXNnumOj+c3652FTqn+au1pC05+//z3p74NFdK0eruh9rz7bymnzjd5nY9mZ7p/qiRjvHLKIbt9NdL0jrM+sz6zPBjBJmoIk/4vuy7FWbyLtV5g3VtUT6H7deEOSA0ZGj7HIdf3St3NveBe6TPtOuiYBm/fi2oiHT9QTWe5tdB+8/rIfoGsqMBl3tphGL+vWCc7/RmAPYN+q2pruUjhA6E5o2yXZZsB8g7ZvjX1C70uf5PF0zTeOobuEvg1wbVvPRPXX+b/p2pM/h65N/K69uMeKb6JWAouT9GN76HOwjs9Z3+hjDJM7NmusN8mWdL9g37aOZd9Gd9y2Gme9n6NrZnFhkof+MaqqG6rqMOCxwNuAjyXZYhLxTsRkj80twJeqapvea8uq+uMJLn/c+avq8qo6hG6bP0nXbGZCcY4z7y3AiaPWuXn7hXRCy9bwsf5Zy1TrH4BtR51jdmnxjehvy5109z48ufe9enRVjfwDupK199lYbgGeOMa4yZxDVtHtu4mud9A6rM+sz6zPGpOk9ZBk6/YrwNl0ba2vGTDNC5L8YjsZ3Et3w9vIZdfb6dpfT9YrkuyZZHPgzcDH2qXc/6H7pea3k2xCd7PqI3vz3Q7sml53saOcBfx5kt3ayeLvgHPapdkJa7F8BDgxyVYtGXkDXRvdidiKrtK5O8l2wPG9Za8EPgO8J8m2STZJMpJE3Q48ZlRzgKuAg5Nsl+QXgNf3xm1B90VaBZDkCLorSetrK7rLvd+n+2fh70aNX9/jDV075geBY5JsnOQQes0C1/E56zsPeFKS/92W8/vAnnQ3yU7UwUmekWRTupuBL62qW8Zbdhv/38DfJ9ksya/S/aL2of6Cq+of6NrsX5hk+7Ztr0iyqLqmLXe3SafcJfEotwM7tW2aiE/Tbesr22dwkyT/K8kvj7P8J0xk/iSbJnl5kkdX1c94+HiOLGf0Z/wh65j3fcBrk+ybzhbtXLFVb9nr+/nULLP+GWwa6p8Rf9u+T8+ka0700THW93O679Y7kjwWIMniJM9rk3wEeHVvnx0/aDnNqcARSQ5I8oi2nF9q4yZ8Dmn74OPACelaZuxJd8/GZFifWZ9ZnzUmSZPzH0nuo8tk/4ruBs8jxph2d+DzdD1sfBV4Tz38HIG/B/463aXC/zOJ9X+Q7ia879HdNPhnAFV1D/AndG1jb6X7Za/fB/3ISf77SQa1CT2tLfti4Ca6my7/dBJx9f1pW/+NdL9wfrgtfyL+me6GxzuBS+iaLvS9ku6Xwm/S3bj3eoDq2m6fBdzY9unj2vZ8na6pwOfoOnygTf8NurbqX6X7Qv0K8JXJbeYazqS75H4rXW8so9san0rXtvbuJJ+czIKr6qd0N7ceSXdifQXdien+Nsl4n7P+cr5PV+G/ka7y+wvgBVV15yTC+TBdRb8aeCrdZfWJLPswul8jbwM+ARxfVRcMiPEtdL8YfT5dknwQcF2SH9DdWHtoVf1kEvFOxBfoel/6XpJ17ovWzOJAuvsObqP7Lr6NNf8p7Hsn3b0CdyV51wTmfyVwc7pmLq+lO95jfcZHG2veZXTtuP+V7gbk5XQ3+I5Y3/ORZpf1z7pNpf6BbtvuovtufojuHtix7g2Crhv25cAl7Xv3ebrWEFTVZ+jqtC+0ab4w1kKq6jK6Y/kOug4cvsTDVzMmew45hq451ffojtcHJr75gPWZ9Zn12UNGeruRNE8kuRR4b1VNtvKTJA2QZH+6K3M7zXUsC4n1mYaZV5KkIZfk2Ul+oV36P5zuptDRV9kkSRpq1meaTzaUpytLG7I96Nq3b0nXA9JL2j1akiTNJ9ZnmjdsbidJkiRJPTa3kyRJkqSeDbK53fbbb1+77rrrXIchSQvaFVdccWdVLVr3lAuP9ZQkDYex6qoNMknaddddWbZs2VyHIUkLWpLRT65XYz0lScNhrLrK5naSpA1akp2TfDHJ9UmuS/K6Vr5dkguS3ND+btvKk+RdSZYnuTrJPr1lHd6mv6H1zjVS/tQk17R53tUeiClJmqdMkiRJG7oHgDdW1S8D+wFHJ9kTOBa4sKp2By5s7wGeT/dgy92Bo4CToUuq6B5AuS/wNOD4kcSqTXNUb76DZmG7JEkzxCRJkrRBq6qVVXVlG74PuB5YDBwCnNEmOwN4URs+BDizOpcA2yTZEXgecEFVra6qu4ALgIPauK2r6qvVdRl7Zm9ZkqR5yCRJkrRgJNkVeApwKbDDyDNa2t/HtskWA7f0ZlvRysYrXzGgfPS6j0qyLMmyVatWTcfmSJJmiEmSJGlBSLIlcC7w+qq6d7xJB5TVepSvWVB1SlUtqaolixbZ6Z8kDTOTJEnSBi/JJnQJ0oeq6uOt+PbWVI72945WvgLYuTf7TsBt6yjfaUC5JGmeMkmSJG3QWk9zpwLXV9U/9UYtBUZ6qDsc+FSv/FWtl7v9gHtac7zzgQOTbNs6bDgQOL+Nuy/Jfm1dr+otS5I0D22Qz0mSJKnn6cArgWuSXNXK/hI4CfhIkiOB7wIvbePOAw4GlgM/Ao4AqKrVSd4CXN6me3NVrW7DfwycDjwK+Ex7SZLmKZMkSdIGraq+zOD7hgAOGDB9AUePsazTgNMGlC8D9ppCmJKkIWKSJGle+/tPL5v1dR73giWzvk5JUucnd9461yGs02bbr9XBpeYZ70mSJEmSpB6TJEmSJEnqMUmSJEmSpB6TJEmSJEnqMUmSJEmSpB6TJEmSJEnqMUmSJEmSpB6TJEmSJEnqMUmSJEmSpB6TJEmSJEnqMUmSJEmSpB6TJEmSJEnqMUmSJEmSpB6TJEmSJEnqmdEkKcmfJ7kuybVJzkqyWZLdklya5IYk5yTZtE37yPZ+eRu/a285x7XybyV53kzGLEmSJGlhm7EkKcli4M+AJVW1F7ARcCjwNuAdVbU7cBdwZJvlSOCuqvpF4B1tOpLs2eZ7MnAQ8J4kG81U3JIkSZIWtplubrcx8KgkGwObAyuB3wI+1safAbyoDR/S3tPGH5Akrfzsqrq/qm4ClgNPm+G4JUkbiCSnJbkjybW9snOSXNVeNye5qpXvmuTHvXHv7c3z1CTXtJYN72p1FEm2S3JBayFxQZJtZ38rJUnTacaSpKq6FXg78F265Oge4Arg7qp6oE22AljchhcDt7R5H2jTP6ZfPmCehyQ5KsmyJMtWrVo1/RskSZqvTqdrifCQqvr9qtq7qvYGzgU+3hv97ZFxVfXaXvnJwFHA7u01ssxjgQtbC4kL23tJ0jw2k83ttqW7CrQb8DhgC+D5AyatkVnGGDdW+ZoFVadU1ZKqWrJo0aL1C1qStMGpqouB1YPGtatBLwPOGm8ZSXYEtq6qr1ZVAWcyuCVEv4WEJGmemsnmds8BbqqqVVX1M7pf6X4D2KY1vwPYCbitDa8AdgZo4x9NV6k9VD5gHkmSpuKZwO1VdUOvbLckX0vypSTPbGWL6eqjEf1WDTtU1UqA9vexMx20JGlmzWSS9F1gvySbt1/qDgC+AXwReEmb5nDgU214aXtPG/+F9mvdUuDQ1vvdbnRNHC6bwbglSQvHYax5FWklsEtVPQV4A/DhJFszwVYN47FZuCTNHzN5T9KldB0wXAlc09Z1CvAm4A1JltPdc3Rqm+VU4DGt/A20Nt1VdR3wEboE67PA0VX14EzFLUlaGFqrhd8Fzhkpa50Efb8NXwF8G3gS3ZWjnXqz91s13N6a4400y7tj0PpsFi5J88fG655k/VXV8cDxo4pvZEDvdFX1E+ClYyznRODEaQ9QkrSQPQf4ZlU91IwuySJgdVU9mOQJdK0Xbqyq1UnuS7IfcCnwKuBf2mwjLSFOYs0WEpKkeWqmuwCXJGlOJTkL+CqwR5IVSUaez3coa3fY8Czg6iRfp2sN8dqqGun04Y+B99M9iuLbwGda+UnAc5PcADy3vZckzWMzeiVJkqS5VlWHjVH+6gFl59J1CT5o+mXAXgPKv093360kaQPhlSRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJ0gYtyWlJ7khyba/shCS3JrmqvQ7ujTsuyfIk30ryvF75Qa1seZJje+W7Jbk0yQ1Jzkmy6extnSRpJpgkSZI2dKcDBw0of0dV7d1e5wEk2RM4FHhym+c9STZKshHwbuD5wJ7AYW1agLe1Ze0O3AUcOaNbI0macSZJkqQNWlVdDKye4OSHAGdX1f1VdROwHHhaey2vqhur6qfA2cAhSQL8FvCxNv8ZwIumdQMkSbNu47kOQJI0M/Z+zbtnfZ1XnXb0rK9zCo5J8ipgGfDGqroLWAxc0ptmRSsDuGVU+b7AY4C7q+qBAdOvIclRwFEAu+yyy3RtgyRpBnglSZK0EJ0MPBHYG1gJ/GMrz4Bpaz3K1y6sOqWqllTVkkWLFk0+YknSrPFKkiRpwamq20eGk7wP+HR7uwLYuTfpTsBtbXhQ+Z3ANkk2bleT+tNLkuYpryRJkhacJDv23r4YGOn5bilwaJJHJtkN2B24DLgc2L31ZLcpXecOS6uqgC8CL2nzHw58aja2QZI0c7ySJEnaoCU5C9gf2D7JCuB4YP8ke9M1jbsZ+COAqrouyUeAbwAPAEdX1YNtOccA5wMbAadV1XVtFW8Czk7yVuBrwKmztGmSpBlikiRJ2qBV1WEDisdMZKrqRODEAeXnAecNKL+Rrvc7SdIGwuZ2kiRJktRjkiRJkiRJPSZJkiRJktRjkiRJkiRJPSZJkiRJktRjkiRJkiRJPSZJkiRJktRjkiRJkiRJPSZJkiRJktQzo0lSkm2SfCzJN5Ncn+TXk2yX5IIkN7S/27Zpk+RdSZYnuTrJPr3lHN6mvyHJ4TMZsyRJkqSFbeMZXv47gc9W1UuSbApsDvwlcGFVnZTkWOBY4E3A84Hd22tf4GRg3yTbAccDS4ACrkiytKrumuHYJWnSXnjSJ2Z1fUuPffGsrk+SpIVgxq4kJdkaeBZwKkBV/bSq7gYOAc5ok50BvKgNHwKcWZ1LgG2S7Ag8D7igqla3xOgC4KCZiluSJEnSwjaTze2eAKwCPpDka0nen2QLYIeqWgnQ/j62Tb8YuKU3/4pWNlb5GpIclWRZkmWrVq2a/q2RJEmStCDMZJK0MbAPcHJVPQX4IV3TurFkQFmNU75mQdUpVbWkqpYsWrRofeKVJEmSpBm9J2kFsKKqLm3vP0aXJN2eZMeqWtma093Rm37n3vw7Abe18v1HlV80g3FLWoeLv3XrrK7vWXusdfFYkiRpxszYlaSq+h5wS5I9WtEBwDeApcBID3WHA59qw0uBV7Ve7vYD7mnN8c4HDkyybesJ78BWJkmSJEnTbqZ7t/tT4EOtZ7sbgSPoErOPJDkS+C7w0jbtecDBwHLgR21aqmp1krcAl7fp3lxVq2c4bkmSJEkL1IwmSVV1FV3X3aMdMGDaAo4eYzmnAadNb3SSJEmStLYZfZisJElzLclpSe5Icm2v7P+1B51fneQTSbZp5bsm+XGSq9rrvb15nprkmvbQ83clSSsf+JB0SdL8ZZIkSdrQnc7az9e7ANirqn4V+B/guN64b1fV3u312l75ycBRPPzg85FlHkv3kPTdgQsZvydXSdI8YJIkSdqgVdXFwOpRZZ+rqgfa20voek4dU+uNdeuq+mprHn4maz4MfdBD0iVJ85RJkiRpoXsN8Jne+93aQ9C/lOSZrWwx3SMpRvQfbD7WQ9LX4EPPJWn+MEmSJC1YSf4KeAD4UCtaCezSHoL+BuDDSbZmgg82H48PPZek+WOmuwCXJGkoJTkceAFwQGtCR1XdD9zfhq9I8m3gSXRXjvpN8kYeeA5jPyRdkjRPeSVJkrTgJDkIeBPwwqr6Ua98UZKN2vAT6DpouLE1o7svyX6tV7tXsebD0Ac9JF2SNE95JUmStEFLchawP7B9khXA8XS92T0SuKD15H1J68nuWcCbkzwAPAi8tvcA8z+m6ynvUXT3MI3cx3QSgx+SLkmap0ySJEkbtKo6bEDxqWNMey5w7hjjlgF7DSj/PgMeki5Jmr9sbidJkiRJPSZJkiRJktRjkiRJkiRJPSZJkiRJktRjkiRJkiRJPSZJkiRJktRjkiRJkiRJPRNKkpJcOJEySZJminWRJGm2jPsw2SSbAZvTPaV8WyBt1NbA42Y4NkmSrIskSbNu3CQJ+CPg9XSV0BU8XDHdC7x7BuOSJGmEdZEkaVaNmyRV1TuBdyb506r6l1mKSZKkh1gXSZJm27quJAFQVf+S5DeAXfvzVNWZMxSXJElrsC6SJM2WCSVJST4IPBG4CniwFRdgxSRJmhXWRZKk2TKhJAlYAuxZVTWTwUiSNA7rIknSrJjoc5KuBX5hJgORJGkdrIskSbNioleStge+keQy4P6Rwqp64YxEJUnS2qyLJEmzYqJJ0gkzGYQkSRNwwlwHIElaGCbau92XZjoQSZLGs751UZLTgBcAd1TVXq1sO+Acup7ybgZeVlV3JQnwTuBg4EfAq6vqyjbP4cBft8W+tarOaOVPBU4HHgWcB7zO+6YkaX6b0D1JSe5Lcm97/STJg0nunengJEkaMYW66HTgoFFlxwIXVtXuwIXtPcDzgd3b6yjg5Lbu7YDjgX2BpwHHJ9m2zXNym3ZkvtHrkiTNMxO9krRV/32SF9FVEpIkzYr1rYuq6uIku44qPgTYvw2fAVwEvKmVn9muBF2SZJskO7ZpL6iq1W3dFwAHJbkI2LqqvtrKzwReBHxm0hsoSRoaE+3dbg1V9Ungt6Y5FkmSJmyKddEOVbWyLWcl8NhWvhi4pTfdilY2XvmKAeVrSXJUkmVJlq1atWo9w5YkzYaJPkz2d3tvH0H3rArbW0uSZs0s1UUZUFbrUb52YdUpwCkAS5YssQ6VpCE20d7tfqc3/ADdTa6HTHs0kiSNbTrrotuT7FhVK1tzujta+Qpg5950OwG3tfL9R5Vf1Mp3GjC9JGkem+g9SUfMdCCSJI1nmuuipcDhwEnt76d65cckOZuuk4Z7WiJ1PvB3vc4aDgSOq6rVrUOJ/YBLgVcB/zKNcUqS5sBEe7fbKcknktyR5PYk5ybZad1zSpI0Pda3LkpyFvBVYI8kK5IcSZccPTfJDcBz23vouvC+EVgOvA/4E4DWYcNbgMvb680jnTgAfwy8v83zbey0QZLmvYk2t/sA8GHgpe39K1rZc2ciKEmSBlivuqiqDhtj1AEDpi3g6DGWcxpw2oDyZcBe48UgSZpfJpokLaqqD/Ten57k9TMRkCRJY7AukqbgnmXnz3UI43r0kufNdQjSQybaBfidSV6RZKP2egXw/ZkMTJKkUayLJEmzYqJJ0muAlwHfA1YCLwHszEGSNJusiyRJs2KiSdJbgMOralFVPZauojphIjO2X/u+luTT7f1uSS5NckOSc5Js2sof2d4vb+N37S3juFb+rSRei5WkhWm96yJJkiZjoknSr1bVXSNvWo8+T5ngvK8Dru+9fxvwjqraHbgLOLKVHwncVVW/CLyjTUeSPYFDgScDBwHvSbLRBNctSdpwTKUukiRpwiaaJD2i92wIkmzHBDp9aF2z/jZd16gkCfBbwMfaJGcAL2rDh7T3tPEHtOkPAc6uqvur6ia6LlafNsG4JUkbjvWqiyRJmqyJVi7/CPx3ko8BRdcm/MQJzPfPwF8AW7X3jwHurqoH2vsVwOI2vBi4BaCqHkhyT5t+MXBJb5n9eSRJC8f61kWSJE3KhJKkqjozyTK6q0ABfreqvjHePEleANxRVVck2X+keNDi1zFuvHn66zsKOApgl112GS80SdI8tD51kSRJ62PCzRRaRTSZyujpwAuTHAxsBmxNd2VpmyQbt6tJOwG3telXADsDK5JsDDwaWN0rH9Gfpx/fKcApAEuWLFkriZIkzanJG5QAABXWSURBVH/rURdJkjRpE70nadKq6riq2qmqdqXreOELVfVy4It03bYCHA58qg0vbe9p47/Qnny+FDi09X63G7A7cNlMxS1JkiRpYZuLG17fBJyd5K3A14BTW/mpwAeTLKe7gnQoQFVdl+QjdL8cPgAcXVUPzn7YkiRJkhaCWUmSquoi4KI2fCMDeqerqp8ALx1j/hPx5lxJkiRJs2DGmttJkiRJ0nxkkiRJkiRJPSZJkiRJktRjkiRJkiRJPXPRu500L/zkzltndX2bbb94VtcnSZKkwbySJEmSJEk9JkmSpAUpyR5Jruq97k3y+iQnJLm1V35wb57jkixP8q0kz+uVH9TKlic5dm62SJI0XWxuJ0lakKrqW8DeAEk2Am4FPgEcAbyjqt7enz7JnnQPOn8y8Djg80me1Ea/G3gusAK4PMnSqvrGrGyIJGnamSRJkgQHAN+uqu8kGWuaQ4Czq+p+4KYky3n44ejL28PSSXJ2m9YkSZLmKZvbSZLUXSE6q/f+mCRXJzktybatbDFwS2+aFa1srPI1JDkqybIky1atWjW90UuSppVJkiRpQUuyKfBC4KOt6GTgiXRN8VYC/zgy6YDZa5zyNQuqTqmqJVW1ZNGiRVOOW5I0c2xuJ0la6J4PXFlVtwOM/AVI8j7g0+3tCmDn3nw7Abe14bHKJUnzkFeSJEkL3WH0mtol2bE37sXAtW14KXBokkcm2Q3YHbgMuBzYPclu7arUoW1aSdI85ZUkSdKClWRzul7p/qhX/A9J9qZrMnfzyLiqui7JR+g6ZHgAOLqqHmzLOQY4H9gIOK2qrpu1jZAkTTuTJEnSglVVPwIeM6rsleNMfyJw4oDy84Dzpj1ASdKcsLmdJEmSJPWYJEmSJElSj0mSJEmSJPWYJEmSJElSj0mSJEmSJPWYJEmSJElSj0mSJEmSJPWYJEmSJElSjw+TleaB2++6b9bXucO2W836OiVJkoaBV5IkSZIkqcckSZIkSZJ6TJIkSZIkqcckSZIkSZJ6TJIkSZIkqcckSZIkSZJ6TJIkSZIkqcckSZIkSZJ6TJIkSQtWkpuTXJPkqiTLWtl2SS5IckP7u20rT5J3JVme5Ook+/SWc3ib/oYkh8/V9kiSpodJkiRpofvNqtq7qpa098cCF1bV7sCF7T3A84Hd2+so4GTokirgeGBf4GnA8SOJlSRpfjJJkiRpTYcAZ7ThM4AX9crPrM4lwDZJdgSeB1xQVaur6i7gAuCg2Q5akjR9TJIkSQtZAZ9LckWSo1rZDlW1EqD9fWwrXwzc0pt3RSsbq1ySNE9tPNcBSJI0h55eVbcleSxwQZJvjjNtBpTVOOVrztwlYUcB7LLLLusTqyRplnglSZK0YFXVbe3vHcAn6O4pur01o6P9vaNNvgLYuTf7TsBt45SPXtcpVbWkqpYsWrRoujdFkjSNZuxKUpKdgTOBXwB+DpxSVe9sN7ieA+wK3Ay8rKruShLgncDBwI+AV1fVlW1ZhwN/3Rb91qo6A0mSpiDJFsAjquq+Nnwg8GZgKXA4cFL7+6k2y1LgmCRn03XScE9VrUxyPvB3vc4aDgSOm8VNkTRP3X7XfXMdwjrtsO1Wcx3CnJjJ5nYPAG+sqiuTbAVckeQC4NV0vQadlORYul6D3sSavQbtS9dr0L69XoOW0DVfuCLJ0nZzrCRJ62sH4BPdb3RsDHy4qj6b5HLgI0mOBL4LvLRNfx7dD3nL6X7MOwKgqlYneQtweZvuzVW1evY2Q5I03WYsSWo3u47c+HpfkuvpbmQ9BNi/TXYGcBFdkvRQr0HAJUlGeg3an9ZrEEBLtA4Czpqp2CVJG76quhH4tQHl3wcOGFBewNFjLOs04LTpjlGSNDdm5Z6kJLsCTwEuxV6DJEmSJA2xGU+SkmwJnAu8vqruHW/SAWWT6jUoybIky1atWrV+wUqSJEla8GY0SUqyCV2C9KGq+ngrttcgSZIkSUNrxpKk1lvdqcD1VfVPvVEjvQbB2r0GvSqd/Wi9BgHnAwcm2bb1HHRgK5MkSZKkaTeTvds9HXglcE2Sq1rZX9J1qWqvQZIkSZKG0kz2bvdlBt9PBPYaJEmSJGlIzUrvdpIkSZI0X5gkSZIkSVKPSZIkSZIk9ZgkSZIkSVKPSZIkSZIk9ZgkSZIkSVKPSZIkSZIk9ZgkSZIkSVKPSZIkSZIk9ZgkSZIkSVKPSZIkSZIk9ZgkSZIkSVKPSZIkaUFKsnOSLya5Psl1SV7Xyk9IcmuSq9rr4N48xyVZnuRbSZ7XKz+olS1PcuxcbI8kafpsPNcBSJI0Rx4A3lhVVybZCrgiyQVt3Duq6u39iZPsCRwKPBl4HPD5JE9qo98NPBdYAVyeZGlVfWNWtkKSNO1MkiRJC1JVrQRWtuH7klwPLB5nlkOAs6vqfuCmJMuBp7Vxy6vqRoAkZ7dpTZIkaZ6yuZ0kacFLsivwFODSVnRMkquTnJZk21a2GLilN9uKVjZW+eh1HJVkWZJlq1atmuYtkCRNJ5MkSdKClmRL4Fzg9VV1L3Ay8ERgb7orTf84MumA2Wuc8jULqk6pqiVVtWTRokXTErskaWbY3E6StGAl2YQuQfpQVX0coKpu741/H/Dp9nYFsHNv9p2A29rwWOWSpHnIK0mSpAUpSYBTgeur6p965Tv2JnsxcG0bXgocmuSRSXYDdgcuAy4Hdk+yW5JN6Tp3WDob2yBJmhleSZIkLVRPB14JXJPkqlb2l8BhSfamazJ3M/BHAFV1XZKP0HXI8ABwdFU9CJDkGOB8YCPgtKq6bjY3RJI0vUySJEkLUlV9mcH3E503zjwnAicOKD9vvPkkSfOLze0kSZIkqcckSZIkSZJ6TJIkSZIkqcckSZIkSZJ6TJIkSZIkqcckSZIkSZJ67AJcQ+OeZefP+jofveR5s75OSZIkDTevJEmSJElSj1eSJEmSxvDtf3vLXIewTk/8o7+Z6xCkDY5XkiRJkiSpxytJC9xs/0Lmr12SJEkadl5JkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQee7ebZV884tBZX+dvfuDsWV+nJEmSNF95JUmSJEmSeubNlaQkBwHvBDYC3l9VJ0103n/75f1mLK5B/uj6S2Z1fZKkuTeVekqSNFzmRZKUZCPg3cBzgRXA5UmWVtU35jYySZKsp0abi6blk2VTdGl6XfytW+c6hHE9a4/Fk5p+vjS3exqwvKpurKqfAmcDh8xxTJIkjbCekqQNSKpqrmNYpyQvAQ6qqj9o718J7FtVx/SmOQo4qr3dA/jWNKx6e+DOaVjOdBmmeIYpFhiueIYpFhiueIYpFhiueIYpFpieeB5fVYumI5hhN4f11LoM2+dqKjaUbdlQtgPclmG0oWwHzN62DKyr5kVzOyADytbI7qrqFOCUaV1psqyqlkznMqdimOIZplhguOIZplhguOIZplhguOIZplhg+OKZB+aknlqXDek4bijbsqFsB7gtw2hD2Q6Y+22ZL83tVgA7997vBNw2R7FIkjSa9ZQkbUDmS5J0ObB7kt2SbAocCiyd45gkSRphPSVJG5B50dyuqh5IcgxwPl3XqqdV1XWzsOpZbRYxAcMUzzDFAsMVzzDFAsMVzzDFAsMVzzDFAsMXz1Cbw3pqXTak47ihbMuGsh3gtgyjDWU7YI63ZV503CBJkiRJs2W+NLeTJEmSpFlhkiRJkiRJPQs+SUpyWpI7klw7xvgkeVeS5UmuTrLPDMayc5IvJrk+yXVJXjfH8WyW5LIkX2/x/O2AaR6Z5JwWz6VJdp2peNr6NkrytSSfHoJYbk5yTZKrkiwbMH42j9U2ST6W5Jvt8/PrcxjLHm2fjLzuTfL6OYznz9vn99okZyXZbNT42f7cvK7Fct3o/dLGz+i+GXTOS7JdkguS3ND+bjvGvIe3aW5Icvh0xqWpGaZz41QM03l1qobpvDwVw3ZOn4phqw+mYq7rkqmYN/VQVS3oF/AsYB/g2jHGHwx8hu4ZGPsBl85gLDsC+7ThrYD/Afacw3gCbNmGNwEuBfYbNc2fAO9tw4cC58zw8XoD8GHg0wPGzXYsNwPbjzN+No/VGcAftOFNgW3mKpZR690I+B7dg9pmPR5gMXAT8Kj2/iPAq+fqcwPsBVwLbE7Xcc7ngd1nc98MOucB/wAc24aPBd42YL7tgBvb323b8Laz8TnyNaHjOjTnxilux9CcV6dhW4byvDzFbZrTc/oUYx+q+mCK2zLndckU458X9dCCv5JUVRcDq8eZ5BDgzOpcAmyTZMcZimVlVV3Zhu8Drqf7Us9VPFVVP2hvN2mv0T19HEJXEQB8DDggyaCHKk5Zkp2A3wbeP8YksxbLBM3KsUqyNd0J51SAqvppVd09F7EMcADw7ar6zhzGszHwqCQb01Uoo59dM5ufm18GLqmqH1XVA8CXgBcPiGfG9s0Y57z+PjgDeNGAWZ8HXFBVq6vqLuAC4KDpikvrbx6eG6dirs5lkzLk5+WpGIZz+lQMU30wFXNel0zFfKmHFnySNAGLgVt671ewduIy7dol3qfQXb2Zs3haE46rgDvoPphjxtO+qPcAj5mhcP4Z+Avg52OMn81YoEsYP5fkiiRHjRdPM1PH6gnAKuADrbnN+5NsMUexjHYocNaA8lmJp6puBd4OfBdYCdxTVZ8bK5ZZ+NxcCzwryWOSbE73S9/Oo6aZi2O1Q1WthO7HGuCxA6aZq8+Q1m3Yzo1TMSzn1aka5vPyVMzpOX0qhrA+mIphrUumYujqIZOkdRv0C8KM9pueZEvgXOD1VXXvXMZTVQ9W1d50T49/WpK95iKeJC8A7qiqK8abbDZi6Xl6Ve0DPB84Osmz5iiejekuW59cVU8Bfkh3qXouYnl4hd0DNV8IfHTQ6NmIp7VpPgTYDXgcsEWSV8xFLABVdT3wNrpfvz4LfB14YK7imaRhjWtBG9Jz41QMy3l1qobyvDwVw3BOn4phqw+mYp7XJVMxq9tkkrRuK1gzO9+JtS/PTpskm9AlSB+qqo/PdTwjWjOBi1j7suZD8bTL149m/OaL6+vpwAuT3AycDfxWkn+fo1gAqKrb2t87gE8ATxsrnmamjtUKYEXvKt/H6CrnuYil7/nAlVV1+4BxsxXPc4CbqmpVVf0M+DjwG2PFMkufm1Orap+qelZbzw1jxdPMxrG6faQZRvt7x4Bp5uTco3UaunPjVAzReXWqhvW8PBXDcE6fiqGrD6ZiSOuSqRi6esgkad2WAq9qvYTsR3d5duVMrKi1ez0VuL6q/mkI4lmUZJs2/Ci6E8w3B8Qz0rvIS4AvVNW0Z/VVdVxV7VRVu9Jd7v9CVY3+BWhWYgFIskWSrUaGgQPpLn+PjmfGj1VVfQ+4JckeregA4BtzEcsohzG4WcZsxvNdYL8km7fv1wF09/qNjmVWPjcASR7b/u4C/C5r76O5OFb9fXA48KkB05wPHJhk2/aL7IGtTHNo2M6NUzFM59WpGuLz8lQMwzl9KoauPpiKIa1LpmL46qEagl4u5vJF96FaCfyMLkM9Engt8No2PsC7gW8D1wBLZjCWZ9BdNrwauKq9Dp7DeH4V+FqL51rg/7byNwMvbMOb0V16Xw5cBjxhFo7Z/rQenOYqFrr25l9vr+uAv2rlc3Ws9gaWtWP1SbpeX+Yklra+zYHvA4/ulc3VvvlbuuT+WuCDwCPn8jMM/BfdP0tfBw6Y7X0zxjnvMcCFdL9EXghs16ZdAry/N+9r2n5aDhwxk/vJ13od2zk/N04x/qE6r07D9gzVeXmK2zI05/QpbsdQ1QdT3JY5rUumGPu8qIfSVihJkiRJwuZ2kiRJkrQGkyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkiRJkqQekyRJkiRJ6jFJkqZBkv2TjH5y92SX8YNpiuUvR73/7+lYriRp/rBOWdt07RMtDD4nSZoGSU4AflBVb5/CMn5QVVtOYLqNqurBqS5HkjS/rOv8P2raeVenJNm4qh6YweVbP2rCvJIkjSPJJ5NckeS6JEe1soOSXJnk60kuTLIr3VOu/zzJVUmemeT0JC/pLecH7e+WbZ4rk1yT5JAJxrF/ki8m+TDdk7PHiu0k4FEtjg+NWvf+SS5K8rEk30zyoSRp4w5uZV9O8q4kn56ePShJmogku7bz8BlJrm7n6s2T3Jzk/yb5MvDSJE9M8tl2/v+vJL/U5t8tyVeTXJ7kLetY19DUKUlOSHJKks8BZybZLMkHWh35tSS/2aZ7dZJ/7c336ST7j8SU5MRWL1+SZIfJ7hNptI3nOgBpyL2mqlYneRRweZJPAe8DnlVVNyXZro1/L70rSUmOHGN5PwFeXFX3JtkeuCTJ0prYJd2nAXtV1U1jxHZuVR2b5Jiq2nuMZTwFeDJwG/AV4OlJlgH/1tumsyYQiyRp+u0BHFlVX0lyGvAnrfwnVfUMgCQXAq+tqhuS7Au8B/gt4J3AyVV1ZpKjJ7CuYapTngo8o6p+nOSNAFX1Ky0B/FySJ61j/i2AS6rqr5L8A/CHwFuZ/D6RHuKVJGl8f5bk68AlwM7AUcDFI5VKVa2e5PIC/F2Sq4HPA4uBHSY472W9ymxQbLtPcBkrqurnwFXArsAvATf2lm2SJElz45aq+kob/nfgGW34HOhaIwC/AXw0yVV0yciObZqn8/D5+4MTWNcw1SlLq+rHbfgZI/FX1TeB7wDrSpJ+CoxcrbqixQGT3yfSQ7ySJI2hXcZ/DvDrVfWjJBcBX6f7pW9dHqD9CNGaH2zayl8OLAKeWlU/S3IzsNkEQ/rhOmKbyHLu7w0/SHcOyATXL0maWaNbFYy8Hzn/PwK4e5wrO5O50XyY6pQf9obHmv+herXpx/ezXouMkThGePO91otXkqSxPRq4q1UYvwTsBzwSeHaS3QCSbNemvQ/YqjfvzXTNBwAOATbpLfOOliD9JvD4aYxtxM+SbDLGfIN8E3hCunurAH5/PWOSJE3NLkl+vQ0fBny5P7Kq7gVuSvJS6H6ES/JrbfRXgEPb8Msnud5hqlMupsXfmtntAnyLrl7dO8kjkuxM11xwXaayT7TAmSRJY/sssHFrGvcWuiYIq+ia3H28NUs4p037H8CL282tz6S7b+nZSS4D9uXhX8k+BCxpbbZfTleZTFdsI04Brh65yXZdWhOHPwE+224Mvh24Zz3jkiStv+uBw9u5fTvg5AHTvBw4stVB19H9EAfwOuDoJJfTJT2TMUx1ynuAjZJcQ1fHvrqq7qdLeG6i62ji7cCVE1jWVPaJFji7AJdEki2r6getaeC7gRuq6h1zHZckLRTtysunq2qvOQ5lyqxTtCHwSpIkgD9sNwFfR/dr27/NcTySpPnLOkXznleSpCGS5FdYuwee+6tq37mIR5I0fw1TnZLkCLrmb31fqSq75tZQMkmSJEmSpB6b20mSJElSj0mSJEmSJPWYJEmSJElSj0mSJEmSJPX8/94LCyZSph3sAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "palette = sns.color_palette(\"RdBu\", 10)\n",
    "fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(14, 4))\n",
    "\n",
    "sns.countplot(x='actual_rating', data=df_pred, palette=palette, ax=ax1)\n",
    "ax1.set_title('Distribution of actual ratings of books in the test set')\n",
    "\n",
    "sns.countplot(x='pred_rating_round', data=df_pred, palette=palette, ax=ax2)\n",
    "ax2.set_title('Distribution of predicted ratings of books in the test set')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Absolute error of predicted ratings\n",
    "\n",
    "The distribution of absolute errors is right-skewed, showing that the majority of errors is small: between 0 and 1. There is a long tail that indicates that there are several observations for which the absolute error was close to 10.\n",
    "\n",
    "How good/bad the model is with predicting certain scores? As expected from the above charts, the model deals very well with predicting score = 8 (the most frequent value). The further the rating from score = 8, the higher the absolute error. The biggest errors happen to observations with scores 1 or 2 which indicates that probably the model is predicting high ratings for those observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAEXCAYAAAB2y3GBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZwcdZ3/8dene47MTGZyTu7JfZADCBACGMQAKiAKuqKiK+quiq736u6KPy+8XXVddZdV8AIRjBiugMihEC4hJIQECEkghNyBTM5J5j4+vz+qJnSGnpnOpLurp/v9fDz6MX3U8e7qmqr+9PdbVebuiIiIiIiI9CexqAOIiIiIiIgcLRUyIiIiIiLS76iQERERERGRfkeFjIiIiIiI9DsqZEREREREpN9RISMiIiIiIv2OCpkcYWa/MLOvpmla483skJnFw8dLzewj6Zh2OL2/mNkH0zW9o5jvt81st5m93Idx07oMwmleaWa/T+c0c124Xk2OOoeIFBZtw9PDzBaY2QvhtvztUecBMLN/NLN7MzTtNWa2MBPTltygQiYLzGyTmTWa2UEz229mfzezj5vZ4eXv7h9392+lOK039jSMu29x94Hu3p6G7K/Z0Lv7Be5+3bFO+yhz1ABfAGa5+6hszjsd8mWHGa5XG492PDObaGZuZkXHmsHMrjWzbx/rdJJMd6GZbUv3dEVyUbgvaTGz4V2eXxX+r06MJlluypdtOPBN4H/Dbflt2Z55sn2Bu9/g7m/OxPzcfba7L+3LuGHOqceaIVPrTjr3q/2ZCpnseZu7VwITgO8DXwR+ne6Z5PEKPQHY4+67og6S65KtA0e7XuTxeiQir3oJeG/nAzM7HiiLLo5AxrfhE4A16cqVZJh4X6Yt0mfurluGb8Am4I1dnpsPdABzwsfXAt8O7w8H7gT2A3uBhwmKzuvDcRqBQ8B/ABMBBz4MbAEeSniuKJzeUuB7wBPAAeB2YGj42kJgW7K8wPlAC9Aazm91wvQ+Et6PAV8BNgO7gN8Bg8LXOnN8MMy2G/hyD8tpUDh+bTi9r4TTf2P4njvCHNcmGXdIuMxqgX3h/XEJr/e0DAYAvwf2hMt8OTAyfG0MsCT8HDYAH02Y5pXA749hOQ4iKGZ3AtuBbwPxbpZNDLgCeDHMeVNC/p7WgcPPhcNeRLAT2x8uk5ld8n4ReBpoJlx/uuRwYGrCOnsV8GfgILAMmNJN/i3huIfC2xnh8/8MrA0/s3uACeHzBvw3wTp1IMw0B7g8XI4t4XTuSDKvpOOGr5UCPwrzvAL8guCLWwVHrmOHgDFRbzt00y1Tt/D//SvA8oTnfgR8OfxfnRg+l/R/Jnwtle3ut4BHw23EvcDwbvJoG57hbXg478TvEKUpLJ/F4bKtI9zvd5nmtcDPgbuA+nB5XQg8FY6zFbgyYfjX7AuADwGPJAzjwMeBF8J14SrAwtfiwH8RfJ94CfgUCd93ulnP35jwfm4i+J5xMFyO87oZ76FwuvVhzveEz78VWBUu/78DJySM88VwPTgIrAfOpZt1J8n8XjNuCutN0v1qod0iD1AIN5IUMuHzW4B/Ce9fy6uFzPcIdhbF4e31Cf/ER0yLVzd2vyP4MlZG8kJmO8EXwQrgZlLYeIf3r+wcNuH1pbxayPwzwcZvMjAQuAW4vku2X4a5TiTYuM7sZjn9jmDnVBmO+zzw4e5ydhl3GPBOoDwc/0/AbV0yd7cMPgbcEY4bB04BqsLXHgT+j2BHOZdgJ3tu12XTx+V4G3B1mGcEwQ76Y928v88BjwPjCHY+VwN/SGEdSHxuOsFG+U0E69V/hJ9dSULeVUAN4ReVJDm6FjJ7CYryIuAGYFE343XmKUp47u3h/GeG438F+Hv42nnAk8BggsJkJjC66/9KN/PqadyfEOy0h4bryR3A91JZx3TTLZ9uvPolfX34PxIn+NI5gSMLmZ7+Z1LZ7r4YbnvKwsff7yaPtuHZ2YYffk8pLp9Wgm11LNk0CbbHB4AF4TADwmV5fPj4BIIC+O1d3mvivuBDvLaQuZNgGz4+zHR++NrHgefC5TgE+GvX6fXyGTYBbyFYT74HPN7D/8jh/V34+GSCH8hOC8f/YDj9UmAGwf/PmIT3OaW7dafLfHoaN5X1Jul7L5SbupZFawfBzqGrVmA0wa/Tre7+sIdrbQ+udPd6d2/s5vXr3f1Zd68Hvgq8O01NwP8I/NjdN7r7IeBLwKVdmqC/4e6N7r4aWE1Q0BwhzPIe4EvuftDdNxH86nJZKiHcfY+73+zuDe5+EPgO8IYug3W3DFoJdqJT3b3d3Z9097rwuJwzgS+6e5O7rwJ+lWqmnpjZSOAC4HPh57aLoBXh0m5G+RhBa9Y2d28m2DBe0mU5J1sHEp97D/Bnd7/P3VsJfmUtA16XMPzP3H1rD+tRV7e4+xPu3kZQyMxNcbzO9/Q9d18bjv9dYK6ZTSD4TCqB4wiK+LXuvjPF6SYd18wM+Cjwr+6+N1xPvkv3y1ykEFwPfIDgy/E6gmIBgN7+Z1Lc7v7W3Z8Ptyk30c02QttwIMvb8BSXz2Pufpu7d/Qwzdvd/dFwmCZ3X+ruz4SPnwb+wGs/y9583933u/sW4AFeXW/eDfw0XI77CLrqH41H3P0uD44hvp4k30d68FHgandfFq5n1xH8OHs60E5QaMwys2J33+TuL6Y43Z7GTWW9KWgqZKI1luAX7a5+SPAry71mttHMrkhhWluP4vXNBL/mDO9m2KMxJpxe4rSLgJEJzyWeZayBoOWmq+FASZJpjU0lhJmVm9nVZrbZzOoImoUHdynWulsG1xN0a1pkZjvM7AdmVhy+t86d91Fn6sWEcP47wxNA7Cf4pWVED8PfmjDsWoKNX+JyTrYOJD53xGfl7h3h62O7GT4VqXy23ZkA/DThPe0laEEZ6+73A/9L0KXgFTO7xsyqUploD+NWE/xi+2TCPO8OnxcpVNcD7yP4Vfx3XV7r8X8mxe1uStsIbcOB7G/DU1k+qUzviGHM7DQze8DMas3sAEErytF+3+huvRnTZX7Hus8acBRFwQTgC52fYfg51hC0pGwgaD25EthlZovMbEwqE+1l3FTWm4KmQiYiZnYqwcbika6vhS0SX3D3ycDbgM+b2bmdL3czyd5abGoS7o8n+AVrN0EzdXlCrjhHfrHrbbo7CP7REqfdRtCUfDR2h5m6Tmt78sFf4wsEzbOnuXsVcFb4vCUMk3QZhK1e33D3WQS/bL2V4BfKHcBQM6tMIdPRLsetBL/kDHf3weGtyt1nd/P+tgIXJAw72N0HuHtilmSfVeJzR3xW4a+tNV3eT2+fd18lm+5Wgm4Yie+pzN3/DuDuP3P3U4DZBF0q/j3VjN2Mu5ugb/jshPkNcvfOHWSm3rtIznL3zQTHGryFoGtwot7+Z1LZ7qZK2/Dsb8NTWT6pTK/rMDcSdEescfdBBF3lrZthj9ZOgm5WnWq6GzADtgLf6fIZlrv7HwDc/UZ3P5NXu2f+ZzheKvus7sbtab3RPgsVMllnZlVm9lZgEUGfyWeSDPNWM5sabqTqCKrvzlMpv0JwPMrRer+ZzTKzcoLTLy4Om1afJ/hF4sLwF6yvEDRxdnoFmGgJp4ru4g/Av5rZJDMbSNDt4I8edBVKWZjlJuA7ZlYZdi/6PMFBhqmoJNjh7jezocDXkwyTdBmY2dlmdny446oj2Dm2u/tWgoP5vmdmA8zsBIIDL29IMu2jWo4edJO6F/ivcJ2ImdkUM+uu+f0X4bKZAGBm1WZ2cYrLptNNwIVmdm6Y8QsEO+K/H+V0+qKW4CDTxHX3F8CXzGw2gJkNMrN3hfdPDX/VKyb4gtFEiv8D3Y0b/nr5S+C/zWxEOOxYMzsvYbrDzGxQet6ySL/xYeCcsMvWYSn8z6Sy3U2VtuG9S+s2/CiXz9GoJGjpaTKz+QQtfp2S7QuOxk3AZ8P1cDDBQfKZ0nVf80vg4+H+xcysIlxfKs1shpmdY2alBPucRo7cZ3X7PaqXcXtab451WeYFFTLZc4eZHSSorr8M/Bj4p26GnUZwANsh4DHg//zV86B/D/hK2Mz4b0cx/+sJDsp7meBgvM8AuPsB4BME/WK3E3zxS7yWxp/Cv3vMbGWS6f4mnPZDBL/qNQGfPopciT4dzn8jQUvVjeH0U/ETgr7CuwkOjLs7yTBJlwEwiuDMLHUEzbYP8moB9V6CA+p2ALcCX3f3+7pOuI/L8QME3emeIzgzy2KCY6OS+SnBL1z3huvR4wQHHKbM3dcD7wf+h2A5vY3gtOAtRzOdvnD3BoI+74+G6+7p7n4rwa9OiyzoSvIsQZ9zgCqCncY+gq4Oewj6g0NwlqBZ4XSSXQehp3G/SNBt8/Fwnn8l+BUYd19HUJhvDKedUrcAkf7O3V909xXdvNzt/wypbXdTpW14LzK0DU9p+RylTwDfDN/n1wiKDyD5vuAop/1LggLyaYIzo91F0AvkmK+bl8SVwHVhzneH/yMfJei6vI/g/+JD4bClBMfr7CZYP0cA/y98rbfvUT2N2+16k4ZlmRc6z4QlIiIiItJvmNkFwC/cfUKvA0teUouMiIiIiOQ8Myszs7eYWZGZjSXognhr1LkkOmqREREREZGcFx4f9SDB6fUbCS7I/Fl3r4s0mERGhYyIiIiIiPQ76lomIiIiIiL9TmRXBh0+fLhPnDgxqtmLiAjw5JNP7nZ3XRQ0Ce2nRESi19N+KrJCZuLEiaxY0d3ZHkVEJBvMbHPvQxUm7adERKLX035KXctERERERKTfUSEjIiIiIiL9jgoZERERERHpd1TIiIiIiIhIv6NCRkRERERE+h0VMiIiUjDMbLCZLTazdWa21szOiDqTiIj0TWSnXxYREYnAT4G73f0SMysByqMOJCIifaNCRkRECoKZVQFnAR8CcPcWoCXKTCIi0nfqWiYiIoViMlAL/NbMnjKzX5lZRdShRESkbwquRWbx0jVJn79k4ewsJxERkSwrAk4GPu3uy8zsp8AVwFc7BzCzy4HLAcaPH3/EyFfPPD17SYGPrX08q/MTEelv1CIjIiKFYhuwzd2XhY8XExQ2h7n7Ne4+z93nVVdXZz2giIikruBaZLqjlhoRkfzm7i+b2VYzm+Hu64FzgeeiziUiIn2jFplutLa1s+qFnbyy91DUUUREJH0+DdxgZk8Dc4HvRpxHRET6SC0ySby0cx93Pf48+w428sBTG/ncu17HO98wm1jMoo4mIiLHwN1XAfOiziEiIsdOLTJdLHtuKzfctxpw3vH6mcyZNJLvXP8g//LjJTQ2t0YdT0REREREUIvMa6xYv52aEYN43xtPoLgozqyJIxg6qJy/PP48H/jOzbxr4WxisZiOnRERERERiVBKLTJmdr6ZrTezDWZ2RZLXP2RmtWa2Krx9JP1RM29vXQP7DjYxe+IIioviAJgZp0wfwwWnTeOFbXv482PP4+4RJxURERERKWy9tsiYWRy4CngTwakrl5vZEnfveqaXP7r7pzKQMWte3LEXgMljh77mtVNmjOVQYwsPP72ZyvJS3nX2nGzHExERERGRUCotMvOBDe6+0d1bgEXAxZmNFY0Xt+9laGUZQyvLkr5+1okTmTt1FI88s5k7Hl2X5XQiIiIiItIplUJmLLA14fG28Lmu3mlmT5vZYjOrSUu6LGprb2fTy/uZkqQ1ppOZccHp05k4ajDfuPYBVqzfnsWEIiIiIiLSKZVCJtk5h7seJHIHMNHdTwD+ClyXdEJml5vZCjNbUVtbe3RJM2zLKwdoa+9gypjuCxmAeCzGO98wm5oRg/j8//6FzS/vz1JCERERERHplEohsw1IbGEZB+xIHMDd97h7c/jwl8ApySbk7te4+zx3n1ddXd2XvBnz4o69xGPG+JGDex22rLSYt5w+nba2Dj70vVv43d2rWLx0TRZSioiIiIgIpFbILAemmdkkMysBLgWWJA5gZqMTHl4ErE1fxOx4cftexo8cTElxPKXhh1SW8a6z51BX38Sflj5LW3tHhhOKiIiIiEinXgsZd28DPgXcQ1Cg3OTua8zsm2Z2UTjYZ8xsjZmtBj4DfChTgTPhQH0Tuw809Hh8TDI1IwbxtgXHsXXXAf6yTKdlFhERERHJlpQuiOnudwF3dXnuawn3vwR8Kb3Rsmf3/gYAxgyrPOpx50waye79DTzyzGZueeg53vkGXShTRERERCTTUrogZr6ra2gCoKpiQJ/GP+vEiUwePYTv3/AQz770SjqjiYiIiIhIEipkgAP1zZhBZXlJn8aPxYy3v34WwwdV8O//dzd19U1pTigiIiIiIolUyAB19c0MLCshHuv74igfUMwPP3Eeu/bV88NFj6YxnYiIiIiIdKVCBqirb2JQH7uVJZozaSQfvvAU7nh0HQ+ueikNyUREREREJBkVMgQtMlXlpWmZ1kffNo/p44bxreuWcuCQupiJiIiIiGRCwRcy7k5dQ3OfD/RPtHjpGm5/ZB1nnTiRvXWNfPqnd+pCmSIiIiIiGVDwhUxDcytt7R1UVaSnRQZg1LBKTp9dw9MvvsK22gNpm66IiIiIiAQKvpCpq28GYFAaCxmAM48fT2VZCfc88QIdHbpQpoiIiIhIOqmQqT+2a8h0p6S4iHNPmcLOPYe47ZG1aZ22iIiIiEihK/hC5kDYIpPOrmWdZk8awbjqKv5n8WPUNTSnffoiIiIiIoWq4AuZuvomiuIxykuL0z5tM+O8+dPYd6iJ3939VNqnLyIiIiJSqFTIhKdeNrOMTH/0sErOmz+VG+5bzZ4DDRmZh4iIiIhIoSn4QuZAfXNGupUl+sQ7TqOltZ1f3bkio/MRERERESkUBV/I1DU0pf1A/64mjBzMxWfO5E9L17B9d11G5yUiIiIiUggKupBp7+jgYENL2k+9nMzHLjqVmBlX37484/MSEREREcl3RVEHiNLBhhYg/ade7mrx0jUAzJ02ijv+vo6aEYMYUlnGJQtnZ3S+IiJyJDPbBBwE2oE2d58XbSIREemrgm6RefUaMplvkQE4ffZ4YmY8tmZLVuYnIiJJne3uc1XEiIj0bwXdIlMXXkMmG13LAKrKSzlx6mhWb9jJmSdMzMo8RUQk/zzwT5dmfZ5n/3ZR1ucpItKTgm6ROdDZIlOe2a5liV43p4YOdx5fszVr8xQRkcMcuNfMnjSzy7u+aGaXm9kKM1tRW1sbQTwREUlVQRcydQ3NlJUUUVIcz9o8Bw8s4/jJo1j5/A721um6MiIiWbbA3U8GLgA+aWZnJb7o7te4+zx3n1ddXR1NQhERSUlhFzJZuIZMMgvmjKetvYNFf3sm6/MWESlk7r4j/LsLuBWYH20iERHpq4IuZOqbWqgYUJL1+Q4bVM70mmH88f5naGxuzfr8RUQKkZlVmFll533gzcCz0aYSEZG+KuhCpqm5jQGlxZHM+4zZ4zlQ38ySR9dFMn8RkQI0EnjEzFYDTwB/dve7I84kIiJ9VNBnLWtsaaWsNJpFMK66ihOmjOT3967ikoWziccKuqYUEck4d98InBh1DhERSY+C/fbs7jS1tDGgJJpCxsz4wHknsXVXHQ+sfCmSDCIiIiIi/VXBFjLNre24Q1lEXcsAzj55EjUjqrju7qdw98hyiIiIiIj0NwVbyDS1BAfZR9UiAxCPxXj/m+fyzMZXWPXCzshyiIiIiIj0NwVbyDQ2twFQVhJdiwzARQuOY/DAAVx3z6pIc4iIiIiI9CcFW8g0tQSFzICIDvbvVFZazLvPmcODq15i0859kWYREREREekvCvasZZ3XbymLsGvZ4qVrAKgoLSFmxjeufYALz5jBJQtnR5ZJRERERKQ/UItMhAf7d6ooK+GEKaN4+sWXOdTYEnUcEREREZGcl1IhY2bnm9l6M9tgZlf0MNwlZuZmNi99ETMjF1pkEp0+q4b2Dmfl8zuijiIiIiIikvN6LWTMLA5cBVwAzALea2azkgxXCXwGWJbukJnQ1NJGPGYUF8WjjgLAsEHlTB07lCfXb6e5tS3qOCIiIiIiOS2VFpn5wAZ33+juLcAi4OIkw30L+AHQlMZ8GdPY3BbpNWSSOW1WDfVNrdy97IWoo4iIiIiI5LRUCpmxwNaEx9vC5w4zs5OAGne/s6cJmdnlZrbCzFbU1tYeddh0amppjfQaMslMHDWY6sEV3HDfal0gU0RERESkB6kUMpbkucPfss0sBvw38IXeJuTu17j7PHefV11dnXrKDGhsyb0WGTNj/sxxPL91DyvW61gZEREREZHupFLIbANqEh6PAxK/ZVcCc4ClZrYJOB1YkusH/Dc1t+VciwzAnEkjGDJwADfcqwtkioiIiIh0J5VCZjkwzcwmmVkJcCmwpPNFdz/g7sPdfaK7TwQeBy5y9xUZSZwmjS2tlEV8McxkioviXLJwDg+u3sSWV/ZHHUdEREREJCf1Wsi4exvwKeAeYC1wk7uvMbNvmtlFmQ6YKU0tbQwoya2uZZ3efc4c4rEYf/jbM1FHERERERHJSSk1Sbj7XcBdXZ77WjfDLjz2WJnV3tFBS2t7TrbIAFQPruD8+dO4/eG1fOLt86ksL406koiIiIhITknpgpj5pqkluE5LLh4j0+l9bzqBhuZWbnt4bdRRRERERERyTu5+k8+gpuagkCnL0a5li5euAaBmxCB+decKSovjxGIxLlk4O+JkIiIiIiK5oSBbZBpbWgEYkKNdyzqdNmscB+qbWb91d9RRRERERERySkEWMrneItNp+rjhDKks49FntugCmSIiIiIiCXK7SSJDGjuPkcnxFplYzFgwZzx3PraeF7fvjTqOiIhIUi9e/a2szm/Kx76a1fmJSG4q0BaZoGtZWQ4f7N/p+Ckjqaoo5eGnN6tVRkREREQkVJCFTGM/OGtZp3gsxoI549m+u47l67ZHHUdEREREJCcUZCHT1NJ6+Exg/cGJU0cxsKyEa+5YoVYZEREREREKtJBpbG7rF60xnYricV43Zzwr1m3nodWboo4jIiIiIhK5gixkmlraKCvN7TOWdXXKjDFMHj2EHy16hJbW9qjjiIj0S2YWN7OnzOzOqLOIiMixKchCprG5tV+1yEBwrMy/v+/1bN1Vx+/vXRV1HBGR/uqzwNqoQ4iIyLEryEKmP7bIAJwxu4azT5rEL+9cwSv7DkUdR0SkXzGzccCFwK+iziIiIseuYAuZ/tYi0+kL71lAe7vzoz88EnUUEZH+5ifAfwAdUQcREZFjV3CFjLvT2NzaL64hk8y4EYP46Nvmcd+KF3lw1UtRxxER6RfM7K3ALnd/spfhLjezFWa2ora2NkvpRESkLwqukGlr76C9wxnQD7uWdfrQBScxZexQvnv9Q9Q3tkQdR0SkP1gAXGRmm4BFwDlm9vuuA7n7Ne4+z93nVVdXZzujiIgchYIrZBqbW4H+cTHMrhYvXcPipWu4/ZF1vP6ECbyy7xCf+5+7oo4lIpLz3P1L7j7O3ScClwL3u/v7I44lIiLHoOAKmaaWNgDKSvtfIZNoXPUg5s0Yw/J123lh256o44iIiIiIZFXBFTKNzUEhM6Ck/3Yt63TWiZMoLY5z1S2PRx1FRKTfcPel7v7WqHOIiMixKbhCprNFpj92LeuqfEAxZ8wez9JVm1i94eWo44iIiIiIZE3BFTLNrUEhU5oHhQzA/JljGVpVxs9ufgx3jzqOiIiIiEhW5Me3+aPQ0toOQGlxPOIk6VFSXMSpM8Zyz/IN/ODGR5gydujh1y5ZODvCZCIiIiIimVNwLTJNnS0yxflTw500fQxVFaX8/dktUUcREREREcmKgitkWlraiMeMonj+vPWieIxTpo9h8yv72b2/Puo4IiIiIiIZlz/f5lPU3NqeN8fHJJo7dTSxmPHk8zuijiIiIiIiknEFV8g0tbblzfExiSrKSpg5vpqnX3z58HFAIiIiIiL5quAKmZbW9rw6PibRKTPG0NzazppNu6KOIiIiIiKSUQVXyDS3tuVtIVMzYhDVg8tZ+fz2qKOIiGSEmcXM7Nmoc4iISPQKr5Bpyc+uZQBmxinTx7JzzyF27jkYdRwRkbRz9w5gtZmNjzqLiIhEKz+bJnrQ3NpOSR4e7N9p9qQR3LdiA89sfCXqKCIimTIaWGNmTwCHT9Xo7hdFF0lERLItf7/Rd6O5tY0BedoiA1BWWsy0ccNY89IrtLV35NVppkVEQt+IOoCIiEQvpW+5Zna+ma03sw1mdkWS1z9uZs+Y2Soze8TMZqU/6rFzd5pb2inJ02NkOh0/eST1Ta0se25r1FFERNLKzOLAV939wa63qLOJiEh29VrIhDuNq4ALgFnAe5MUKje6+/HuPhf4AfDjtCdNg+bWdjrc8/YYmU5Txw6jrKSIOx9bH3UUEZG0cvd2oMHMBkWdRUREopVK08R8YIO7bwQws0XAxcBznQO4e13C8BWApzNkutQ3tgAwII+PkQGIx2PMmjiCB1a+RH1jCxVlJVFHEhFJpybgGTO7jyOPkflMdJFERCTbUvlGPxZI7KO0DTit60Bm9kng80AJcE6yCZnZ5cDlAOPHZ/+EM4fCQibfu5YBHD9lJE8+v4O/rdzIRQuOizqOiEg6/Tm8iYhIAUvlGBlL8txrWlzc/Sp3nwJ8EfhKsgm5+zXuPs/d51VXVx9d0jToLGTyvWsZwNjhVdSMGMQdj66LOoqISFq5+3XATcDj7n5d5y3qXCIikl2pFDLbgJqEx+OAHT0Mvwh4+7GEypT6prCQyfOuZRBcU+aiBcexfN12tu06EHUcEZG0MbO3AauAu8PHc81sSbSpREQk21L5Rr8cmGZmk4DtwKXA+xIHMLNp7v5C+PBC4AVy0KGGwmmRAbhowXH8323LWPLoOj7xjtf0BhQR6a+uJDh+cymAu68K91EiHFhxT9bnOWjeeVmfp4ik0CLj7m3Ap4B7gLXATe6+xsy+aWadFx/7lJmtMbNVBMfJfDBjiY/Boc4WmQI4RgZg5NCBvG72eG5/ZB3tHR1RxxERSZc2d+/a1JyTJ5kREZHMSekbvbvfBdzV5bmvJdz/bJpzZUR9Y2EVMgAXv34m//Hze1j23DZeNyf7J1gQEcmAZ83sfUDczKYBnwH+HnEmERHJsoK67HshHezfaeHcSQweOHd7+lwAACAASURBVIDbHl4bdRQRkXT5NDAbaAZuBA4An4s0kYiIZF3BFTJF8RjxeOG87ZLiOG85fToPPLWRfQcbo44jInLM3L3B3b/s7qeGt6+4e1Pn62b2P1HmExGR7Cicb/QEZy0rpNaYTv9w1ixa2zq4+cE1UUcREcmGBVEHEBGRzCuoQuZQY0tBHR/Taeq4YZwxu4ZFf3uGltb2qOOIiIiIiByzwitkSgqvRQbgsvPmsvtAA39Z9nzUUUREREREjllBNU/UN7ZQUkAtMouXvtqVzN0ZMbiCq25ZxkULjsPMIkwmIpJRSTdwZjYAeAgoJdj/LXb3r2czmIiIpE9BtcjUN7YwoIAKmURmxmmza9i1v57H1myNOo6ISFqYWczMqro8/dNuBm8GznH3E4G5wPlmdnpGA4qISMYUVCFzsLGFkgI82L/TnIkjGFhWwnV/eSrqKCIifWZmN5pZlZlVAM8B683s3ztfd/drk43ngUPhw+Lwpgtpioj0UwVVyARnLSvMFhmAeDzG6bNrWLZ2Gyuf3xF1HBGRvprl7nXA2wku1jweuCyVEc0sbmargF3Afe6+rMvrl5vZCjNbUVtbm+7cIiKSRgVTyLg79QV8sH+nU6aPYfigcn5+2xNRRxER6atiMysmKGRud/dWUmxZcfd2d58LjAPmm9mcLq9f4+7z3H1edXV12oOLiEj6FEzzRFNLG+0dXtAtMgDFRXFOnj6Ge5dv4EeLHmHiqCGHX7tk4ewIk4mIpOxqYBOwGnjIzCYAdUczAXffb2ZLgfOBZ9MdUEREMq9gWmQONbYAFOQFMbs6efpoKstKeHDVJtzVPVxE+hd3/5m7j3X3t4THvWwGzu5tPDOrNrPB4f0y4I3AugzHFRGRDCmYQqa+s5ApKewWGYCieJwFx09g664DvLRzX9RxRESOipkNM7OfmdlKM3vSzH4KDEph1NHAA2b2NLCc4BiZOzMaVkREMqZgCpmDh1tkVMgAzJ02msryUh5arVYZEel3FgG1wDuBS8L7f+xtJHd/2t1PcvcT3H2Ou38zwzlFRCSDCqaQqVfXsiMUxWOcecIEttXWsXHH3qjjiIgcjaHu/i13fym8fRsYHHUoERHJrsIpZJrUtayruVNGMaiiVMfKiEh/84CZXRpeDDNmZu8G/hx1KBERya6CKWQONqhFpqt4PMaZJ0xkx56DbNi+J+o4IiI9MrODZlYHfAy4EWgOb4uAf40ym4iIZF/BFDKHW2R0jMwRTpgyksEDB/Dgqk10dKhVRkRyl7tXunuVu1cCw4EzCc48djbwtkjDiYhI1hVOIaNjZJKKx2KcdeJEXt57iPtWbIg6johIr8zsI8CDwN3AleHfr0WZSUREsq9gCplDjS0MKCkiFiuYt5yyOZNGUj24nKtuXUZrW3vUcUREevNZ4FRgs7ufDZwE7I42koiIZFvBfKs/1NjCwLKSqGPkpFjMOPukyWx55QBLHtW14UQk5zW5exOAmZW6+zpgRsSZREQkywqmkKlvUiHTk2njhnHClFH84vblNLW0RR1HRKQn28xsMHAbcJ+Z3Q7siDiTiIhkWcEUMocaW6hQIdMtM+Mzl5xO7f56/nj/M1HHERHplru/w933u/uVwFeBXwNvjzaViIhkW0EVMgMHqJDpybwZY1kwZzy/+fOTHGxojjqOiEiv3P1Bd1/i7i1RZxERkewqmEKmXi0yKfn0O0/nQH0zv7t7VdRRRERERES6VTAXVTnY0EJluQqZnixeugaAWROrufbulVSUlTCwrIRLFs6OOJmIiIiIyJEKpkXmYEMzleWlUcfoFxbOnURbewePPrM56igiIiIiIkkVRCHT1t5BQ3OrCpkUDa0qZ+7U0Tz5/A72HWyMOo6IiIiIyGsURCFzqDE4BrRSx8ik7PUnTiRmxkOrN0UdRURERETkNQqikOk8A5daZFJXVV7KqceN5ZmNr7Bh256o44iIiIiIHCGlQsbMzjez9Wa2wcyuSPL6583sOTN72sz+ZmYT0h+171TI9M3r5oyntDjO/966LOooIiIiIiJH6LWQMbM4cBVwATALeK+Zzeoy2FPAPHc/AVgM/CDdQY+FCpm+KSst5ow541n61Eus3vBy1HFERERERA5LpUVmPrDB3TeGFxxbBFycOIC7P+DuDeHDx4Fx6Y15bA42hMfI6PTLR23+ceMYUlnGL+9cEXUUEREREZHDUilkxgJbEx5vC5/rzoeBvyR7wcwuN7MVZraitrY29ZTH6FCjWmT6qqQ4zj++6QQeeXoz67fsjjqOiIiIiAiQWiFjSZ7zpAOavR+YB/ww2evufo27z3P3edXV1amnPEavtsiokOmLd59zPBUDivntXSujjiIiIiIiAkBRCsNsA2oSHo8DdnQdyMzeCHwZeIO7N6cnXnocbGgmZkZ5aXHUUfqle5/YwAlTRnHP8heYPGYIQ6vKAbhk4eyIk4mIiIhIoUqlkFkOTDOzScB24FLgfYkDmNlJwNXA+e6+K+0pj9HBhmYGlpcQiyVrXJJUzJ85jifWbuPx57byltNnRB1HRESkX2javT2r8xswvKfe/yL5pdeuZe7eBnwKuAdYC9zk7mvM7JtmdlE42A+BgcCfzGyVmS3JWOI+ONjQQmWZupUdi8ryUk6cOorVG14+fIFREREREZGopNIig7vfBdzV5bmvJdx/Y5pzpdXBxmadsSwNTptZw8rnd/Lk+u28Ye6kqOOIiIiISAFL6YKY/d3BhmYd6J8GwwaVM3XsUJ58fgdt7e1RxxERERGRAlYghUyLCpk0OW1WDQ1NrTy7MecOhRIR6ZGZ1ZjZA2a21szWmNlno84kIiJ9l1LXsv4uaJFR17J0mDhqMCMGV/DE2m24O2Y6gYKI9BttwBfcfaWZVQJPmtl97v5c1MFEROToFUiLjLqWpYuZMX/mOHbtr+eJtduijiMikjJ33+nuK8P7BwlOYKNTPImI9FN5X8i0tXdQ39SqQiaN5kweQcWAYn5/7+qoo4iI9ImZTQROApZ1ef5yM1thZitqa2ujiCYiIinK+0KmPjxVcGWZupalS1E8zsnTx/Dw05vZtHNf1HFERI6KmQ0EbgY+5+51ia+5+zXuPs/d51VXV0cTUEREUpL3hczBhmYAtcik2SkzxlJcFOOGvz4ddRQRkZSZWTFBEXODu98SdR4REem7/C9kOltkVMik1cCyEt5y+nTueHQdBw41RR1HRKRXFpyd5NfAWnf/cdR5RETk2OR/IaMWmYx5/5vn0tTSxs0Prok6iohIKhYAlwHnmNmq8PaWqEOJiEjf5P3pl18tZHSMTLpNGzeM02aOY9H9z3DZeXMpLopHHUlEpFvu/gigc8aLiOQJtcjIMbnsvLns2lfPnx9bH3UUERERESkgBVDI6BiZTFpw/HhmTajm139+krb2jqjjiIiIiEiBKIBCppmYGeWlxVFHyTuLl67h5gefY/akEWzdVce3rnsg6kgiIiIiUiAKopCpKCshFlO36EyZXjOcEYMreOSZLbR3qFVGRERERDKvAA72b9GB/hlmZpx5wgRueeg5/rriRc6bPy3qSCIiIiKSBnP/+aqsz3PVbz6Z0nB53yJzqLFZx8dkwXHjqxk+qJyf3/YErW3tUccRERERkTyX94VM0CKjQibTYjHjnJMns+nl/dzy0HNRxxERERGRPFcAhUwzlWXqWpYN08YNY95xY/nFbU8cPu21iIiIiEgm5H8ho65lWWNmfOE9C9h3qInf3rUy6jgiIiIiksfyv5BR17KsmjmhmreeMYPf37ua7bV1UccRERERkTyV14VMe0cHhxp11rJsWrx0DZPHDAHgsz/7M3964FkWL10TcSoRERERyTd5XcjUN7YAqEUmy6oqBvCGuRPZsH0vazfXRh1HRERERPJQXl9H5mCDCpmonHrcWJ7Z+Ar3Lt/A5DFDo44jIiJS8F7ZdzDr8xw5pDLr85TCkdctMp1nzlLXsuyLxWK85fTp1De1cP/KjVHHEREREZE8UyCFjFpkojBmeBXzjxvHyud38PiarVHHEREREZE8kueFTNi1rEyFTFQWnjSJYVVlXPnb+3VtGRERERFJm7wuZA7UNwFQVaFCJirFRXEuOnMmu/bV88M/PBJ1HBERERHJE3ldyOw+0ADAsKryiJMUtrHDq/jnC09myaPr+NuTL0YdR0RERETyQF4XMnvqGqiqKKWkOB51lIL3sYtOZdbEEVz52wfYuSf7Z00RERERkfyS14XM7gMNDFdrTE4oLorz/Y+9ifb2Dr78y/toa++IOpKIiIiI9GMpFTJmdr6ZrTezDWZ2RZLXzzKzlWbWZmaXpD9m3+w50MCwQSpkcsX4kYP58gfewMrnd/LLO1ZEHUdERERE+rFeL4hpZnHgKuBNwDZguZktcffnEgbbAnwI+LdMhOyr3QfqmTN5ZNQxBFi8dM3h+8dPHsk1dyynobmVL7xnQYSpRERERKS/SqVFZj6wwd03unsLsAi4OHEAd9/k7k8DOdVfSF3LctP5p01j8MAybnt4LfsPNUUdR0RERET6oVQKmbFA4tUMt4XP5bSGphYam9vUtSwHlRYX8Q9nzaK+qYVv/PZ+3D3qSCIiIiLSz6RSyFiS5/r0zdPMLjezFWa2ora2ti+TSFnnqZeHD6rI6Hykb0YPq+SckyfzwFMvcdMDz0YdR0RERET6mVQKmW1ATcLjccCOvszM3a9x93nuPq+6urovk0jZns5ryKhFJmedNnMcC44fz38tepQXtu2JOo6I5Dkz+42Z7TIz/XoiIpIHUilklgPTzGySmZUAlwJLMhvr2O2u62yRUSGTq8yMb334XKoqSvniz++hsbk16kgikt+uBc6POoSIiKRHr4WMu7cBnwLuAdYCN7n7GjP7ppldBGBmp5rZNuBdwNVmtqb7KWbH7v1hi4wO9s9p9698iTefOpWNO/fxiR/fweKla444w5mISLq4+0PA3qhziIhIevR6+mUAd78LuKvLc19LuL+coMtZzth9oIGieIzBAwdEHUV6MXnMUM6YXcNja7YybdwwptcMjzqSiBQoM7scuBxg/PjxEacRkUJ10fdvzer8llzxjqzOL11SuiBmf7TnQANDq8qIxZKdq0ByzcK5kxg5pII/P7ae+qaWqOOISIHK5rGcIiJybPK2kNldp2vI9CfxeIyLz5xJU0sbdz3+vE7JLCIiIiI9yttCZs+BBp2xrJ8ZMWQgC0+axPotu7n9kbVRxxERERGRHJa3hczuAw06Y1k/dNrMGiaMGsz3b3iYF7frmFwRSR8z+wPwGDDDzLaZ2YejziQiIn2Xl4VMe0cHe+vUItMfxWLG28+cSVlpMV/8hU7JLCLp4+7vdffR7l7s7uPc/ddRZxIRkb7Ly0LmwKEm2jtcLTL9VGV5Kd/56BvZsH0vP7jxYR0vIyIiIiKvkZeFzO4DnRfDrIg4ifTV6+aM58MXnsKtD6/l9/eujjqOiIiIiOSYlK4j0990FjLqWta/ffIdp7H55f381x8fZeTQgbz51KlRRxIRERGRHJGXhcyewy0yKmT6s1jM+PZH30jtgXq+8su/MrSqjHkzxkYdS0RERERyQF4WMrvrwhaZqrKIk0hfLV665vD9c0+ewrbaOj7+oyX8z+feyhmzayJMJiIiIiK5ID8Lmf0NlJcWUz6gJOookgblA4q57M1zufGvq/nMT+/kR584nzfMnRR1LBERETkGD63fntX5naVeHXknLw/218Uw88/AshIue/NcptcM5wtX3c09T7wQdSQRERERiVBeFjK76xqoHqxCJt+UlRZz9b9dzAlTRvKlq+/jtofXRh1JRERERCKSl13L9hxoYOrYoVHHkAy4e9kLvGneVPYfbOLK397PY89u4dSZ47hk4eyoo4mIiIhIFuVdi0x7Rwc79xxkxJCBUUeRDCkuivPuc45nRs1w7lm+gaVPvaSLZoqIiIgUmLwrZDa/vJ+mljZmjB8edRTJoKJ4jHe+YRZzp47ikWc2863rltLW3hF1LBERERHJkrzrWrZuy24AjlMhk/disRgXnjGDirISbnnoOXbuOch//st5VJWXRh1NRERERDIs71pk1m2ppaQozqTRQ6KOIllgZpx90mS+/qGzWb5uOx/8zs1s3XUg6lgiIiIikmH51yKzuZap44ZRXBSPOopk0TvOmsW4EYP4t6v+wnu/cRNf/eBCzps/LepYIiIi0g98784VWZ/nl946L+vzzDd51SLj7qzbslvdygrQ4qVr2Pzyfi47by6DBw7gi7+4lw9+92bqGpqjjiYiIiIiGZBXhcyOPQepq29m5oTqqKNIRAYPLOOy8+ay4PjxrN7wMhd/6QZufeg5Ojp0VjMRERGRfJJXhcz6zTrQXyAei3H2SZP58IWnMH7kIL5x7QO86+uLWLz0WRqaWqKOJyIiIiJpkFfHyKzbUkvMjKnjhkUdRXLA6GGVXPulf+DuZS9w7d1P8e3fPchP/vQYp8+u4fRZ45g7dTTjRw6mpFjHU4mIiIj0N3lVyKzdXMukMUMoKy2OOorkiJsffA6AS94wm221dazasJNlz23jryteBCBmxpjhlYweVsnIIQMZOXQgo4YOPHx/5NCBDKooxcyifBsiIiIi0kVeFTLrtuxm/syxUceQHGRm1IwYRM2IQbg7e+sa2bnnIHvqGg7fX791NwcbmvEuh9OUFsc5fsoo5kwawexJI5g9cQSjh1WquBERERGJUN4UMnsONFC7v57jxutAf+mZmTFsUDnDBpW/5rWODqe+qYW6+mbqGpqoq29mb10jDU2t/P7e1bS1dwAwtKqM4yePZM6kkZwwZSSzJ41kYFlJtt+KiIiISMHKm0Jm3ZZaAI7TGcvkGMRiRmV5KZXlpYyl6ojX2to72LXvEDt2H2THnjqefWkXD67aBIAZTBo9hBnjhzO9ZjjTxw1jes1whg8qV8uNiIiISAbkTSHzyNObiceMGTpjmWRIUTzGmOFVjBleBQRdGBubW9mx5yDba+vYsbuOR5/ewl8ef+HwOEMGDmDCqCFMGDWI8SMHM2HkYCaMGkzNiEEMKMmbfz8RERGRrMuLb1I7dtex+ME1XLTgOKrKS6OOIwWkrLSYKWOGMmXM0MPPNTa3smtfPbv2HWLX/nr21DWwYfseDjUeeernUUMHUjNiEKOGVTJ66MDg77BKRgyuYPDAAVRVlFJcpDOqiYiIiCSTF4XM1UuWYxgfu/jUqKOIUFZazIRRQctLoubWNvbWNbK3roG9BxvZE55kYN2W3RxqfO1JBgCKi2KUFhdRWhyn5PDfOANKihhQUsSgigEMqSxLuL36eFBFUAyVFsfVvU1ERETyTr8vZF7cvpc7Hl3PP77pBEYNrYw6jki3SouLGB22unTV3tHBwYZmDtQ3c6ihmcbmNhpbWmlt66CtvZ229o7g1hb8PdjQwt66RjZs30tjUyuNLW3dzre4KEZleSlV5aVUVQxgWFUZQ6vKGFZVztCq8sP3O5+vLNfppkVERCT3pVTImNn5wE+BOPArd/9+l9dLgd8BpwB7gPe4+6b0Rn2t5tY2fvKnv1NWWsQ/X3hKpmcnkjHxWIzBA8sYPLCsT+N3dHTQ0NxGQ1MLDc2tNITFTVNzK80tbTS1ttHU3EZdfRM79xw8PFx3rUBDK4MCZ2hVGVUVpQwsK6WyrISBZSVUhH8ry0opLysmHosRjxmxmBGPxYiZEY8ZZsFz7o47OOHMHBwOPw/ByRJKiuIUF8cpjgetTiVFMYqL4hTFYyqsJG1625+JiEj/0WshY2Zx4CrgTcA2YLmZLXH35xIG+zCwz92nmtmlwH8C78lEYIDWtnZue3gtv7rzSV7Zd4jPvusMhlT27QugSD6IxWIMDAuMVHV0eFj0tHCosYX6puB+fWMLh8L7G3fspbG5jebWNppb2ulIVvlkmBmHi5viojglRXFKioMi5/DjsAgqKYpTHBZAwXCvDlNcFDtiuCOGPTxuOM7h+cVenU78yHFjMRVX/U2K+zMREeknUmmRmQ9scPeNAGa2CLgYSNzwXwxcGd5fDPyvmZl7Zr71/ODGh/nT0jWcOHUU3/rIucyfOS4TsxHJa7GYHS5+RgzpfXh3p62943BR09zaRnNrUNx4eOvoCFtaCO/jBF/3jcRGleC+Ef7BHdrbO2jv6KCtw4P77R20dXTQ3u60d4SvtXvC8x20dzj1jS0cSBiuLXz+8PTCx53XAEqXonjQEhW8nyOLms6H1vnuOx+/Zrgur3d9vssEkw03a+IIfvbZC4/pvRSQVPZnIiLST1hvtYaZXQKc7+4fCR9fBpzm7p9KGObZcJht4eMXw2F2d5nW5cDl4cMZwPp0vZGjMBzY3etQouWUGi2n1Gg5pS7by2qCuxfEBbhS3J9lYj+VS+t/LmWB3MqTS1kgt/LkUhZQnp7kUhZIT55u91OptMgk6z/RtfpJZRjc/RrgmhTmmTFmtsLd50WZoT/QckqNllNqtJxSp2WVUb3uqzKxn8qlzzSXskBu5cmlLJBbeXIpCyhPT3IpC2Q+TyyFYbYBNQmPxwE7uhvGzIqAQcDedAQUERFJk1T2ZyIi0k+kUsgsB6aZ2SQzKwEuBZZ0GWYJ8MHw/iXA/Zk6PkZERKSPUtmfiYhIP9Fr1zJ3bzOzTwH3EJyu8jfuvsbMvgmscPclwK+B681sA0FLzKWZDH2MIu3a1o9oOaVGyyk1Wk6p07LKkO72Z1mYdS59prmUBXIrTy5lgdzKk0tZQHl6kktZIMN5ej3YX0REREREJNek0rVMREREREQkp6iQERERERGRfqdgChkzO9/M1pvZBjO7Iuo8ucrMaszsATNba2ZrzOyzUWfKVWYWN7OnzOzOqLPkMjMbbGaLzWxduF6dEXWmXGRm/xr+zz1rZn8wswFRZ5LUmdlvzGxXeF21ZK+bmf0s3Ac9bWYnZzBLr9vxLOcZYGZPmNnqMM83kgxTamZ/DPMsM7OJmcoTzq/b7XcEWTaZ2TNmtsrMViR5PZufVY/b6yxnmREuk85bnZl9LsI8PW6jI1hvPhtmWdN1uYSvZ3TZJNvmmdlQM7vPzF4I/ya93LaZfTAc5gUz+2CyYVLmCVflztcbwUGdLwKTgRJgNTAr6ly5eANGAyeH9yuB57Wsul1WnwduBO6MOksu34DrgI+E90uAwVFnyrUbMBZ4CSgLH98EfCjqXLod1Wd4FnAy8Gw3r78F+AvBtWxOB5ZlMEuv2/Es5zFgYHi/GFgGnN5lmE8AvwjvXwr8McOfV7fb7wiybAKG9/B6Nj+rHrfX2czSZb5x4GWCCyNmPU8q2+hsrjfAHOBZoJzgxF1/BaZlc9kk2+YBPwCuCO9fAfxnkvGGAhvDv0PC+0P6mqNQWmTmAxvcfaO7twCLgIsjzpST3H2nu68M7x8E1hL8A0sCMxsHXAj8KuosuczMqgg2dr8GcPcWd98fbaqcVQSUWXAtrnJ0fZN+xd0foufrp10M/M4DjwODzWx0hrKksh3PZh5390Phw+Lw1vVMQxcTfIkGWAyca2bJLmB6zFLYfmctS4qy8lmluL3O2nrTxbnAi+6+OcI8vW2js7nezAQed/cGd28DHgTekSRPxpZNN9u8xGVwHfD2JKOeB9zn7nvdfR9wH3B+X3MUSiEzFtia8Hgb+nLeq7BZ9CSCX8/kSD8B/gPoiDpIjpsM1AK/Dbtx/MrMKqIOlWvcfTvwI2ALsBM44O73RptK0iyS/VAP2/Gs5gm7cq0CdhF8iek2T/jF7AAwLENxett+ZzMLBEXdvWb2pJld3lOeUKY+q1S211F9n7oU+EOS57OSJ8VtdDbXm2eBs8xsmJmVE7S+1HQZJorPaqS774TgBxVgRJJh0pqrUAqZZBWxzjvdAzMbCNwMfM7d66LOk0vM7K3ALnd/Muos/UARQdPzz939JKCeoLlZEoT9iC8GJgFjgAoze3+0qSTNsr4f6mU7ntU87t7u7nOBccB8M5sTRZ4Ut9/Z/qwWuPvJwAXAJ83srIjypLK9jmI9LgEuAv6U7OVs5ElxG521ZePua4H/JGjNuJvgkIm2qPIcpbTmKpRCZhtHVqrjULeNbplZMcHO7wZ3vyXqPDloAXCRmW0i6KZ4jpn9PtpIOWsbsC3h19fFBDtKOdIbgZfcvdbdW4FbgNdFnEnSK6v7oRS245HsF8OuSkt5bVeSw3nCrjuD6LmrXl+lsv3OVhYA3H1H+HcXcCtBd/ikeUKZ+qxS2V5Hsd5cAKx091eSvJatPKlso7O93vza3U9297PC+bzQXZ5QNj6rVzq7r4V/dyUZJq25CqWQWQ5MM7NJYWV/KbAk4kw5KezP+Wtgrbv/OOo8ucjdv+Tu49x9IsG6dL+769fzJNz9ZWCrmc0InzoXeC7CSLlqC3C6mZWH/4PnEhzXIPljCfCB8ExCpxN0TdmZiRmluB3PZp5qMxsc3i8j+FK4LkmezrMXXUKwXU37r8cpbr+zkgXAzCrMrLLzPvBmgm5DXfNk/LNKcXudtfUmwXtJ3q0sm3lS2UZnbb0BMLMR4d/xwD/w2mUUxWeVuAw+CNyeZJh7gDeb2ZCwpevN4XN9UtTXEfsTd28zs08RLKg48Bt3XxNxrFy1ALgMeCbszwzw/9z9rggzSf/2aeCG8EeEjcA/RZwn57j7MjNbDKwk6B7wFHBNtKnkaJjZH4CFwHAz2wZ8neCgdtz9F8BdBP3YNwANZPb/IOl2HBgfUZ7RwHVmFif4AfUmd7/TzL4JrHD3JQSF1/VmtoHg1+VLM5jnNSLMMhK4NTwmvAi40d3vNrOPQySf1Wu21xFmITz+403AxxKey3qe7rbREa/DN5vZMKAV+KS778vmsvn/7d1dqO1zHsfx94dzGOPISYwLeSxRx0yJQnnYgySm0TSkcEGM5LG5IKmZhOSC1NRcoKScjRmaRkwZDznpHOl4Og9zTkcKF1KejqeNwgKKUwAABF5JREFUToyvi/VbWpa1WWdb27L2fr9u9v//+/9/v//3vy7Wd3//v99/71m+824B/pnkQjrF31nt3COBS6rqoqramuRGOpMMADdU1ZxnrjKPxaIkSZIkzYvFsrRMkiRJ0gJiISNJkiRp4ljISJIkSZo4FjKSJEmSJo6FjCRJkqSJYyEjSZIkaeJYyEg9ksyMOwZJ0uKUZCpJ/3+M394xRpLHklzXt//sKMaVRslCRvqJJVnSt7/juGKRJP2sTAE/qpAZ1hC551uFTFX9JHFJ28NCRotWkn8neTHJpiQX97TfluSlJE8l2au1XZlkc5INSR74njF3TXJ3kueTvJzkjNZ+fpIHkzwCPN6euj2d5D5g43zfqyRpfAblmySntlyzvuWbA4BLgD8nWZfkuCT3JDmzZ5yZ9nNZ6/NSko3dXDNEHN/JPbPEdguwS4tjuu/aU0lWJXkoyZYk00nSjp3W2lYn+VuSR0fzCUqDparGHYM0Fkn2qKqtSXYBngdOAN4Dzquq6SR/BX5VVZcneQs4sKq2JVleVR/OMubNwOaqWplkObAWOBw4C7gJ+E275hTwH+Cwqnp93m9WkjQ2A/LNScALwPFV9XrP8euBmaq6tfW7B3i0qh5q+zNVtazN7P+yqj5OsifwHHBwVVX3nFnimKIv9wzKhVX1fv84PdeeAh4GVgBvAWuAq9v9vNpzT/cDu1XV70b3SUrf5oyMFrMrk6ynkwD2BQ4GvgL+0Y6vBI5t2xuA6STnAV9+z5inANcmWQesAn4B7NeOPVFVW3vOXWsRI0mLQn++uRh4ppsD+nLDMALcnGQD8CSwD7D3kH37c8+gXDjMGG9W1VfAOuAA4FDgtZ6x7x8yHmnOlvzwKdLC054onQwcU1WfJVlFp+jo152yPB04Hvg98JckK6pqUEET4I9V9Urf9Y4CPu07t39fkrTAzJJv1gOHDNH9S9pD57Z8a6fWfi6wF3BEVX2R5A0G57BBvsk925EL+23r2f4/nd8nM+T1pZFxRkaL1e7AB+2L+1Dg6Na+A9Bdj3wOsDrJDsC+VfU0cA2wHBg4bQ/8F7iiZ73w4fN1A5KkiTAo3+wMnJDkQOgs72rnfgLs1tP3DeCItn0GsLRnzHdaEfNbYP8Rxtb1RZKls/QbZAtwUHvXB+DsOcYkDc1CRovVY8CSNi1/I50pdeg8qVqR5EXgROAGYEdgZZKNwMvA7bO9I9PGWgpsSPK/ti9JWrwG5Zt36Swv+1db1tVd0vwI8Ifuy/7AXXQKnrVA78z+NHBkkhfozM5sGWFsXXfSyWXTwwxUVZ8DlwKPJVkNvA18NMe4pKH4sr8kSZJ+tCTLqmqmrUr4O/BqVd0+7ri0cDkjI0mSpFH4U/tjN5voLFu7Y8zxaIFzRkaagyQXAFf1Na+pqsvGEY8kSV1Jfg3c29e8raqOGkc80nyxkJEkSZI0cVxaJkmSJGniWMhIkiRJmjgWMpIkSZImjoWMJEmSpInzNRMh18b3NxeTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_pred_err = df_pred.groupby('actual_rating')['abs_err'].mean().reset_index()\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(14, 4))\n",
    "\n",
    "sns.distplot(df_pred['abs_err'], color='#2f6194', ax=ax1)\n",
    "ax1.set_title('Distribution of absolute error in test set')\n",
    "\n",
    "sns.barplot(x='actual_rating', y='abs_err', data=df_pred_err, palette=palette, ax=ax2)\n",
    "ax2.set_title('Mean absolute error for rating in test set')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis of predicted ratings of a particular user\n",
    "\n",
    "For this part of the analysis, the user with id 193458 was selected. By analyzing book ratings by this user, it can be noted that he/she likes diverse types of readings: English romantic novels (Pride and Prejudice, Sense and Sensibility), fantasy (Narnia) as well as historical novels (Schindler's List). Among the recommended books there are other works from Narnia's series, two historical novels and one romance which correlates with user's previous preferences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_books = pd.read_csv('data/books.csv')\n",
    "\n",
    "df_ext = df.merge(df_books[['isbn', 'book_title']], on='isbn', how='left')\n",
    "df_ext['book_title_short'] = df_ext['book_title'].apply(f.short_title)\n",
    "df_ext = df_ext.merge(df_pred[['isbn', 'user_id', 'pred_rating']], on=['isbn', 'user_id'], how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>isbn</th>\n",
       "      <th>book_rating</th>\n",
       "      <th>book_title</th>\n",
       "      <th>book_title_short</th>\n",
       "      <th>pred_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>124989</th>\n",
       "      <td>193458</td>\n",
       "      <td>1853260002</td>\n",
       "      <td>10</td>\n",
       "      <td>Pride &amp;amp; Prejudice (Wordsworth Classics)</td>\n",
       "      <td>Pride &amp;amp; Prejudice (Wordsworth Classics)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124942</th>\n",
       "      <td>193458</td>\n",
       "      <td>0140620125</td>\n",
       "      <td>9</td>\n",
       "      <td>Wuthering Heights (Penguin Popular Classics)</td>\n",
       "      <td>Wuthering Heights (Penguin Popular Classics)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124952</th>\n",
       "      <td>193458</td>\n",
       "      <td>0345342569</td>\n",
       "      <td>9</td>\n",
       "      <td>Shoeless Joe</td>\n",
       "      <td>Shoeless Joe</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124940</th>\n",
       "      <td>193458</td>\n",
       "      <td>0140298479</td>\n",
       "      <td>9</td>\n",
       "      <td>Bridget Jones: The Edge of Reason</td>\n",
       "      <td>Bridget Jones: The Edge of Reason</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124991</th>\n",
       "      <td>193458</td>\n",
       "      <td>1853260169</td>\n",
       "      <td>10</td>\n",
       "      <td>Sense and Sensibility (Wordsworth Classics)</td>\n",
       "      <td>Sense and Sensibility (Wordsworth Classics)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124978</th>\n",
       "      <td>193458</td>\n",
       "      <td>0671880314</td>\n",
       "      <td>9</td>\n",
       "      <td>Schindler's List</td>\n",
       "      <td>Schindler's List</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124951</th>\n",
       "      <td>193458</td>\n",
       "      <td>0330352695</td>\n",
       "      <td>9</td>\n",
       "      <td>Four Letters of Love</td>\n",
       "      <td>Four Letters of Love</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124932</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471047</td>\n",
       "      <td>9</td>\n",
       "      <td>The Lion, the Witch, and the Wardrobe (The Chr...</td>\n",
       "      <td>The Lion, the Witch, and the Wardrobe (The</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124938</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471101</td>\n",
       "      <td>9</td>\n",
       "      <td>The Magician's Nephew (rack) (Narnia)</td>\n",
       "      <td>The Magician's Nephew (rack) (Narnia)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124936</th>\n",
       "      <td>193458</td>\n",
       "      <td>006447108X</td>\n",
       "      <td>9</td>\n",
       "      <td>The Last Battle</td>\n",
       "      <td>The Last Battle</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id        isbn  book_rating  \\\n",
       "124989   193458  1853260002           10   \n",
       "124942   193458  0140620125            9   \n",
       "124952   193458  0345342569            9   \n",
       "124940   193458  0140298479            9   \n",
       "124991   193458  1853260169           10   \n",
       "124978   193458  0671880314            9   \n",
       "124951   193458  0330352695            9   \n",
       "124932   193458  0064471047            9   \n",
       "124938   193458  0064471101            9   \n",
       "124936   193458  006447108X            9   \n",
       "\n",
       "                                               book_title  \\\n",
       "124989        Pride &amp; Prejudice (Wordsworth Classics)   \n",
       "124942       Wuthering Heights (Penguin Popular Classics)   \n",
       "124952                                       Shoeless Joe   \n",
       "124940                  Bridget Jones: The Edge of Reason   \n",
       "124991        Sense and Sensibility (Wordsworth Classics)   \n",
       "124978                                   Schindler's List   \n",
       "124951                               Four Letters of Love   \n",
       "124932  The Lion, the Witch, and the Wardrobe (The Chr...   \n",
       "124938              The Magician's Nephew (rack) (Narnia)   \n",
       "124936                                    The Last Battle   \n",
       "\n",
       "                                    book_title_short  pred_rating  \n",
       "124989   Pride &amp; Prejudice (Wordsworth Classics)          NaN  \n",
       "124942  Wuthering Heights (Penguin Popular Classics)          NaN  \n",
       "124952                                  Shoeless Joe          NaN  \n",
       "124940             Bridget Jones: The Edge of Reason          NaN  \n",
       "124991   Sense and Sensibility (Wordsworth Classics)          NaN  \n",
       "124978                              Schindler's List          NaN  \n",
       "124951                          Four Letters of Love          NaN  \n",
       "124932    The Lion, the Witch, and the Wardrobe (The          NaN  \n",
       "124938         The Magician's Nephew (rack) (Narnia)          NaN  \n",
       "124936                               The Last Battle          NaN  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_user_id = 193458\n",
    "df_user = df_ext[df_ext['user_id']==selected_user_id]\n",
    "\n",
    "df_user[(df_user['pred_rating'].isna())&(df_user['book_rating']>=9)].sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train set: Top rated books\n",
    "\n",
    "![](img/train_actual.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>isbn</th>\n",
       "      <th>book_rating</th>\n",
       "      <th>book_title</th>\n",
       "      <th>book_title_short</th>\n",
       "      <th>pred_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>124946</th>\n",
       "      <td>193458</td>\n",
       "      <td>0142001740</td>\n",
       "      <td>9</td>\n",
       "      <td>The Secret Life of Bees</td>\n",
       "      <td>The Secret Life of Bees</td>\n",
       "      <td>8.281881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124935</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471071</td>\n",
       "      <td>9</td>\n",
       "      <td>The Voyage of the Dawn Treader (rack) (Narnia)</td>\n",
       "      <td>The Voyage of the Dawn Treader (rack) (Narnia)</td>\n",
       "      <td>8.244509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124937</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471098</td>\n",
       "      <td>9</td>\n",
       "      <td>The Silver Chair</td>\n",
       "      <td>The Silver Chair</td>\n",
       "      <td>8.184727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124974</th>\n",
       "      <td>193458</td>\n",
       "      <td>0553258001</td>\n",
       "      <td>9</td>\n",
       "      <td>The Cider House Rules</td>\n",
       "      <td>The Cider House Rules</td>\n",
       "      <td>8.057183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124958</th>\n",
       "      <td>193458</td>\n",
       "      <td>0345431057</td>\n",
       "      <td>9</td>\n",
       "      <td>Slaves in the Family (Ballantine Reader's Circle)</td>\n",
       "      <td>Slaves in the Family (Ballantine Reader's</td>\n",
       "      <td>8.055557</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id        isbn  book_rating  \\\n",
       "124946   193458  0142001740            9   \n",
       "124935   193458  0064471071            9   \n",
       "124937   193458  0064471098            9   \n",
       "124974   193458  0553258001            9   \n",
       "124958   193458  0345431057            9   \n",
       "\n",
       "                                               book_title  \\\n",
       "124946                            The Secret Life of Bees   \n",
       "124935     The Voyage of the Dawn Treader (rack) (Narnia)   \n",
       "124937                                   The Silver Chair   \n",
       "124974                              The Cider House Rules   \n",
       "124958  Slaves in the Family (Ballantine Reader's Circle)   \n",
       "\n",
       "                                      book_title_short  pred_rating  \n",
       "124946                         The Secret Life of Bees     8.281881  \n",
       "124935  The Voyage of the Dawn Treader (rack) (Narnia)     8.244509  \n",
       "124937                                The Silver Chair     8.184727  \n",
       "124974                           The Cider House Rules     8.057183  \n",
       "124958       Slaves in the Family (Ballantine Reader's     8.055557  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_user[df_user['pred_rating'].notna()].sort_values('pred_rating', ascending=False).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test set: predicted top rated books\n",
    "\n",
    "![](img/test_pred.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>isbn</th>\n",
       "      <th>book_rating</th>\n",
       "      <th>book_title</th>\n",
       "      <th>book_title_short</th>\n",
       "      <th>pred_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>124934</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471063</td>\n",
       "      <td>9</td>\n",
       "      <td>The Horse and His Boy</td>\n",
       "      <td>The Horse and His Boy</td>\n",
       "      <td>7.814202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124935</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471071</td>\n",
       "      <td>9</td>\n",
       "      <td>The Voyage of the Dawn Treader (rack) (Narnia)</td>\n",
       "      <td>The Voyage of the Dawn Treader (rack) (Narnia)</td>\n",
       "      <td>8.244509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124937</th>\n",
       "      <td>193458</td>\n",
       "      <td>0064471098</td>\n",
       "      <td>9</td>\n",
       "      <td>The Silver Chair</td>\n",
       "      <td>The Silver Chair</td>\n",
       "      <td>8.184727</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124946</th>\n",
       "      <td>193458</td>\n",
       "      <td>0142001740</td>\n",
       "      <td>9</td>\n",
       "      <td>The Secret Life of Bees</td>\n",
       "      <td>The Secret Life of Bees</td>\n",
       "      <td>8.281881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124958</th>\n",
       "      <td>193458</td>\n",
       "      <td>0345431057</td>\n",
       "      <td>9</td>\n",
       "      <td>Slaves in the Family (Ballantine Reader's Circle)</td>\n",
       "      <td>Slaves in the Family (Ballantine Reader's</td>\n",
       "      <td>8.055557</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        user_id        isbn  book_rating  \\\n",
       "124934   193458  0064471063            9   \n",
       "124935   193458  0064471071            9   \n",
       "124937   193458  0064471098            9   \n",
       "124946   193458  0142001740            9   \n",
       "124958   193458  0345431057            9   \n",
       "\n",
       "                                               book_title  \\\n",
       "124934                              The Horse and His Boy   \n",
       "124935     The Voyage of the Dawn Treader (rack) (Narnia)   \n",
       "124937                                   The Silver Chair   \n",
       "124946                            The Secret Life of Bees   \n",
       "124958  Slaves in the Family (Ballantine Reader's Circle)   \n",
       "\n",
       "                                      book_title_short  pred_rating  \n",
       "124934                           The Horse and His Boy     7.814202  \n",
       "124935  The Voyage of the Dawn Treader (rack) (Narnia)     8.244509  \n",
       "124937                                The Silver Chair     8.184727  \n",
       "124946                         The Secret Life of Bees     8.281881  \n",
       "124958       Slaves in the Family (Ballantine Reader's     8.055557  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_user[df_user['pred_rating'].notna()].sort_values('book_rating', ascending=False).head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test set: actual top rated books\n",
    "\n",
    "![](img/test_actual.jpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "master",
   "language": "python",
   "name": "master"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}