examples/quickstart/rec/rec_synthetic_data_collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quickstart Example with Synthetic Recommender Simulation and Dataset (Data Collection)\n",
    "This notebook provides an example of visualizing the logged dataset collected on an recommender environment.\n",
    "\n",
    "This example on consists of the following case:\n",
    "1. Discrete Action Case\n",
    "2. Collecting Logged Datasets with Multiple Behavior Policies and Random Seeds\n",
    "\n",
    "\\* This library uses [d3rlpy](https://github.com/takuseno/d3rlpy)'s algorithm implementations.  \n",
    "\\* Also, our data collection module is highly inspired by [Open Bandit Pipeline](https://github.com/st-tech/zr-obp)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import SCOPE-RL modules\n",
    "import scope_rl\n",
    "from scope_rl.dataset import SyntheticDataset\n",
    "from scope_rl.policy import OnlineHead\n",
    "from scope_rl.policy import EpsilonGreedyHead\n",
    "from scope_rl.ope.online import (\n",
    "    calc_on_policy_policy_value,\n",
    "    visualize_on_policy_policy_value,\n",
    ")\n",
    "\n",
    "# import recgym\n",
    "from recgym import RECEnv\n",
    "\n",
    "# import d3rlpy algorithms\n",
    "from d3rlpy.algos import DiscreteRandomPolicyConfig\n",
    "\n",
    "# import from other libraries\n",
    "import gym\n",
    "import torch\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.utils import check_random_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "from dataclasses import dataclass\n",
    "from typing import Tuple, Union, Optional\n",
    "from glob import glob\n",
    "from tqdm import tqdm\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1.2\n"
     ]
    }
   ],
   "source": [
    "# version\n",
    "print(scope_rl.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set random state\n",
    "random_state = 12345\n",
    "random_ = check_random_state(random_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = \"cuda:0\" if torch.cuda.is_available() else \"cpu\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ignore warnings\n",
    "import warnings\n",
    "warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Description of Synthetic Recommender Simulation Environment\n",
    "\n",
    "To begin with, we briefly describe the basic usage of the environment.\n",
    "\n",
    "#### RL setup for Recommendation\n",
    "In recommendation, the objective of the RL agent is to maximize reward\n",
    "\n",
    "We often formulate this recommendation problem as the following (Partially Observable) Markov Decision Process ((PO)MDP):\n",
    "- `state`: \n",
    "   - A vector representing user preference.  The preference changes over time in an episode by the actions presented by the RL agent.\n",
    "   - When the true state is unobservable, the agent uses observation instead of state.\n",
    "- `action`: Indicating which item to present to the user.\n",
    "- `reward`: User engagement signal. Either binary or continuous.\n",
    "\n",
    "Let's see how it works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# setup environment\n",
    "env = gym.make('RECEnv-v0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = OnlineHead(\n",
    "    DiscreteRandomPolicyConfig().create(device=device),\n",
    "    name=\"random\",\n",
    ")\n",
    "agent.build_with_env(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# interact agent with the environment\n",
    "# only 6 lines are needed for RL interaction\n",
    "for episode in range(10):\n",
    "    obs, info = env.reset()\n",
    "    done = False\n",
    "\n",
    "    while not done:\n",
    "        action = agent.sample_action_online(obs)\n",
    "        obs, reward, done, truncated, info = env.step(action)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5,)\n",
      "[ 0.13726447  0.0194582   0.10000844 -0.04762143  0.09446818]\n"
     ]
    }
   ],
   "source": [
    "# state \n",
    "print(obs.shape)\n",
    "print(obs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAGwCAYAAABFFQqPAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXP0lEQVR4nO3dd3zV1f3H8dfNHpDBSEIgTNl7hmUBQXH8UKp1UFRUiqOgItqfoK04qqitq2pVcPanFm2t1okiilT2lr0UiEAIEMiEJNx7f38ckpAysu7NueP9fDzuIyc39+a+Lyj55Hw/5xyH2+12IyIiIhKEQmwHEBEREbFFhZCIiIgELRVCIiIiErRUCImIiEjQUiEkIiIiQUuFkIiIiAQtFUIiIiIStMJsB/B1LpeLvXv3Ur9+fRwOh+04IiIiUgVut5u8vDxSU1MJCTnzvI8KoUrs3buXtLQ02zFERESkBjIyMmjWrNkZv65CqBL169cHzB9kXFyc5TQiIiJSFbm5uaSlpZX9HD8TFUKVKL0cFhcXp0JIRETEz1TW1qJmaREREQlaKoREREQkaKkQEhERkaClHiEPcTqdlJSU2I4RdMLDwwkNDbUdQ0RE/JQKoVpyu91kZmZy5MgR21GCVkJCAikpKdrnSUREqk2FUC2VFkFJSUnExMToh3EdcrvdFBYWkpWVBUCTJk0sJxIREX+jQqgWnE5nWRHUsGFD23GCUnR0NABZWVkkJSXpMpmIiFSLmqVrobQnKCYmxnKS4Fb6568eLRERqS4VQh6gy2F26c9fRERqSoWQiIiIBC0VQiIiIhK0VAiJT5s/fz4Oh0PbE4iIiFeoEBKR4FVyDNxu2ylExCIVQkJxcbHtCD6RQYJM5nr4czv46Le2k4iIRSqEPMnthuICO7dq/FY7dOhQJk2axOTJk2nUqBEjR45k/fr1XHTRRdSrV4/k5GSuu+46Dh48CMCnn35KQkICTqcTgDVr1uBwOJg6dWrZ9/zNb37DtddeC8ChQ4cYM2YMTZs2JSYmhq5du/L3v/+90gwAn3/+Oe3atSM6Opphw4axc+fO2vyNiJzZV7+HohzY9Am4XLbTiIgl2lDRk0oK4bFUO699316IiK3yw9966y1uu+02Fi5cyJEjRzjvvPP4zW9+wzPPPMPRo0e59957ueqqq/jmm28499xzycvLY/Xq1fTp04fvvvuORo0aMX/+/LLv991333HvvfcCcOzYMXr37s29995LXFwcn332Gddddx1t2rShX79+p80AkJGRweWXX87EiRO5+eabWbFiBXfffbdn/nxETrZ9Hvz4rRkX58Hhn6BhG7uZRMQKFUJBqm3btjz55JMA/PGPf6Rnz5489thjZV9//fXXSUtLY+vWrbRr144ePXowf/58+vTpw/z587nrrrt46KGHyM/PJycnh+3btzNkyBAAmjZtyj333FP2vW6//Xa+/PJL3n///QqF0MkZAO677z7atGnDU089BUD79u1Zt24dTzzxhFf/LCTIuJwwd3rF+/atVSEkEqRUCHlSeIyZmbH12tXQu3fvsvHatWv59ttvqVev3imP27FjB+3atWPIkCHMnz+fu+++m//85z/MmDGD999/n++//57s7GxSU1Np27YtYI4eeeyxx3j//ffZs2cPxcXFFBUVnbID98kZADZt2kR6enqF+wYMGFCt9yVSqR/eh/3rIDIe2gyFjf+GzB+gy+W2k4mIBSqEPMnhqNblKZtiY8tz5ufnM2rUqNPOvJQeZDp06FBef/111q5dS3h4OB06dGDo0KHMnz+fw4cPl80GAfzpT3/iueee49lnn6Vr167ExsYyefLkUxqiT84gUidKjsE3fzTjc++CqHhTCO1bazeXiFijQkjo1asXH3zwAS1btiQs7PT/SZT2CT3zzDNlRc/QoUN5/PHHOXz4cIVenoULF3LZZZeVNU+7XC62bt1Kp06dzpqjY8eOfPzxxxXuW7JkSW3emkhFy16B3J8hrimk3wpZG839+34wCw50XItI0NGqMWHixIlkZ2czZswYli9fzo4dO/jyyy+58cYby1aKJSYm0q1bN9555x2GDh0KwC9+8QtWrVrF1q1bK8wItW3blrlz57Jo0SI2bdrELbfcwv79+yvNceutt7Jt2zZ+97vfsWXLFt59913efPNNb7xlCUaF2bDA9J9x3u8hPBqSOoMjFAoPQq6ly9oiYpUKISE1NZWFCxfidDq54IIL6Nq1K5MnTyYhIYGQkPL/RIYMGYLT6SwrhBo0aECnTp1ISUmhffv2ZY/7/e9/T69evRg5ciRDhw4lJSWF0aNHV5qjefPmfPDBB3z00Ud0796dl19+uUIDt0it/Ocps1w+qTN0u9rcFx4FjU/8t5v5g71sImKNw+3Wtqpnk5ubS3x8PDk5OcTFxVX42rFjx/jpp59o1aoVUVFRlhKK/h6kUod3wQt9wFkMYz+AtiPKv/bhrbD27zD0Phh6r72MIuJRZ/v5fTLNCIlI4Pvmj6YIajUEzhle8Wsp3cxHNUyLBCUVQiIS2PaugXXvm/H5D5/aEN2ku/moS2MiQUmFkIgELrcb5v7BjLteBak9Tn1MSlfzMSfDNFSLSFBRIeQBarOyS3/+ckbb58FPCyA0wqwUO52oOGjQ2ox1eUwk6KgQqoXw8HAACgsLLScJbqV//qV/HyLAiaM0HjDjfjdDYoszP1Z9QiJBSxsq1kJoaCgJCQlkZWUBEBMTg0MbstUZt9tNYWEhWVlZJCQkEBoaajuS+JIf3oOsDWb36HMrOby3STfY+JH6hESCkAqhWkpJSQEoK4ak7iUkJJT9PYgAUHL0pKM07oaYBmd/fGnDtGaERIKOCqFacjgcNGnShKSkJEpKSmzHCTrh4eGaCZJTLX0ZcvdAfBr0u6Xyx6ecKIQO7YCifIg89QBiEQlMKoQ8JDQ0VD+QRXxBYTb85xkzHna/2T26MvUaQ/1UyNsL+9dD8/7ezSgiPkPN0iISWBb82RylkdwVul1V9ec1UcO0SDBSISQigePwTlg204zPfwhCqjFLW9YnpIZpkWCiQkhEAse8R8BVAq2HnXqURmW0hF4kKKkQEpHAsGcVrP8n4DCzQdVVemnswCY4XuTRaCLiu1QIiYj/c7vLN0/sdlX5Za7qiE+D6ERwHYesjZ7NJyI+S4WQiPi/7V/Dzv+c/SiNyjgcJ10eU5+QSLBQISQi/u3kozTSb4GE5jX/XjqJXiToqBASEf+29u/mUlZUQuVHaVRGO0yLBB0VQiLiv4oL4ZtHzfgX95gen9oomxFab2aaRCTgqRASEf+19CWzG3R8GvSdUPvv16ANhMfC8aNwcFvtv5+I+DwVQiLinwoOwffPmvF5f6jaURqVCQmBlC5mrD4hkaCgQkhE/NOCP0FRLqR0ha5Xeu77qk9IJKioEBIR/5P9Iyx/1YzPf8TM5HiKdpgWCSoqhETE/5QepdHmPGgzzLPf++Ql9G63Z7+3iPgcFUIi4l/2rIQN/wIcMKIGR2lUpnEHCAmHYzlwZJfnv7+I+BQVQiLiP9xu+OrE5ondryk/H8yTwiIguZMZa4dpkYDnd4XQiy++SMuWLYmKiiI9PZ1ly5ZV6XmzZ8/G4XAwevRo7wYUEe/Z9hXs+h5CI2HY/d57HfUJiQQNvyqE3nvvPaZMmcL06dNZtWoV3bt3Z+TIkWRlZZ31eTt37uSee+7h3HPPraOkIuJxzuPlR2n0vxUS0rz3WjpqQyRo+FUh9PTTTzNhwgRuvPFGOnXqxMsvv0xMTAyvv/76GZ/jdDoZO3YsDz30EK1bt67DtCLiUWvfhQObzVEag+/y7mtpCb1I0PCbQqi4uJiVK1cyYsSIsvtCQkIYMWIEixcvPuPzHn74YZKSkhg/fnyVXqeoqIjc3NwKNxGxrLgQvn3MjH/xu9ofpVGZ5M6AA/L3Q95+776WiFjlN4XQwYMHcTqdJCcnV7g/OTmZzMzM0z7n+++/57XXXmPWrFlVfp0ZM2YQHx9fdktL8+L0u4hUzZK/Qt4+c7J8Pw8cpVGZiFho1M6MdXlMJKD5TSFUXXl5eVx33XXMmjWLRo0aVfl506ZNIycnp+yWkZHhxZQiUqmCgycdpfEAhEXWzeuWrkjbt6ZuXk9ErAizHaCqGjVqRGhoKPv3V5ym3r9/PykpKac8fseOHezcuZNRo0aV3edyuQAICwtjy5YttGnT5pTnRUZGEhlZR//QikjlvnsSivNM306XK+rudZt0h3X/0BJ6kQDnNzNCERER9O7dm3nz5pXd53K5mDdvHgMGDDjl8R06dGDdunWsWbOm7HbppZcybNgw1qxZo0teIv7g0A5Y8ZoZn/+wZ4/SqIyW0IsEBb+ZEQKYMmUK48aNo0+fPvTr149nn32WgoICbrzxRgCuv/56mjZtyowZM4iKiqJLly4Vnp+QkABwyv0i4qO+eQRcx+GcEdB6aN2+dkpX8/HILjh62PsN2iJihV8VQldffTUHDhzggQceIDMzkx49ejBnzpyyBurdu3cTUpe/MYqI9/y8AjZ8iNeO0qhMTAPTnH1kN2Sug1a/qPsMIuJ1DrdbpwqeTW5uLvHx8eTk5BAXF2c7jkhwcLvhzUtg10LoMRZG/9VOjtljYfOncMGjMHCSnQwiUiNV/fmt6RMR8T1b55giKCwKht1nL0eTHuajltCLBCwVQiLiW5zH4esHzTj9VohvZi9LEzVMiwQ6FUIi4lvWvGOO0ohO9P5RGpUpPWrj4Fazu7WIBBwVQiLiO4oLTjpK438hOsFqHOqnQGwSuF2wf4PdLCLiFSqERMR3LP4r5GdCQgvoW7XzAb2u7CR6XR4TCUQqhETEN+QfgIXPmvHwOjxKozLqExIJaCqERMQ3LHgSivPNSq3Ol9tOU65sh2mtHBMJRCqERMS+QztgxetmfMEjdXuURmVKL41lbQRnid0sIuJxPvSvjYgErXkPmaM02l7gezs4J7aEyHhwFpvVbCISUFQIiYhdGcth47/BEWLnKI3KOBwn9Qnp8phIoFEhJCL2uN0w9w9m3OPXkNzJbp4z0Un0IgFLhZCI2LPlC9i92BylMdTiURqVKVtCrxkhkUCjQkhE7HAeh6+nm3H/30J8U7t5zqb00ljmOnC57GYREY9SISQidqz+P3N0RXQDGDzZdpqza9jWzFoV50P2j7bTiIgHqRASkbpXXADzZ5jxkHshKt5unsqEhkFyFzPet8ZqFBHxLBVCIlL3Fr8I+fvN0vQ+N9lOUzVll8fUJyQSSFQIiUjdys+Chc+Z8fAHICzCbp6qKm2Y1hJ6kYCiQkhE6tZ3T5hem9Re0OmXttNU3clL6N1uu1lExGNUCIlI3Tm4DVa8YcbnP+xbR2lUJqkThITB0WzI3WM7jYh4iB/9KyQifm/eQ+B2QrsLodW5ttNUT3gUNO5gxtpYUSRgqBASkbqRsQw2fXLiKI0HbaepGZ1ELxJwVAiJiPe53fBV6VEaYyGpo908NVXWMK0ZIZFAoUJIRLxv82eQsQTComGYDx+lURktoRcJOCqERMS7nCXlR2kMmAhxqXbz1EZKV/Mxdw8UHLSbRUQ8QoWQiHjXqr/Boe0Q0xAG3Wk7Te1E1ocGbcxYl8dEAoIKIRHxnqJ8mP+4GQ+5F6Li7ObxBJ1ELxJQVAiJiPcsfgEKsiCxFfS+0XYaz2hy0saKIuL3VAiJiHfk7YeFfzHjEdP95yiNymgJvUhAUSEkIt7x3eNQUgBNe0On0bbTeE7ppbHsHXAs124WEak1FUIi4nkHt8HKt8z4/EfA4bCbx5NiG0FcUzPev95uFhGpNRVCIuJ5Xz944iiNi6DlINtpPE8n0YsEDBVCIuJZu5fA5k/9+yiNyqSoYVokUKgQEhHPOfkojZ7XQVIHu3m8RUvoRQKGCiER8ZxNn8DPyyA8BoZOs53Ge0qX0GdtgpJjdrOISK2oEBIRz3CWwLyHzHjAJIhrYjePN8U1hegGpg8qa6PtNCJSCyqERMQzVr114iiNRjDoDttpvMvh0En0IgFChZCI1F5RXvlRGkOnmjO5Ap1OohcJCCqERKT2Fj0PBQfMgaS9b7Cdpm5oCb1IQFAhJCK1k5dpCiEwR2mEhtvNU1dSThRC+9eD87jdLCJSYyqERKR25j8OJYXQrC90vNR2mrrToDVE1IPjx+DQNttpRKSGVAiJSM0d2AKr/mbGgXaURmVCQiClqxmrYVrEb6kQEpGa+/ohs4S8/SXQYoDtNHVPJ9GL+D0VQiJSM7sWw5bPwBEauEdpVEZL6EX8ngohEak+txvmnjhKo9f10Lid3Ty2nLyE3uWym0VEakSFkIhU36aP4eflEB4b2EdpVKZxBwiNgKJcOLLTdhoRqQEVQiJSPc4S+PpBMx44CeonW41jVWg4JHUyY/UJifglFUIiUj0r34TsHyG2MQy83XYa+3QSvYhfUyEkIlV3LDf4jtKoTGmfkBqmRfySCiERqbpFz0PhQWh4DvQaZzuNb0g5aeWY2203i4hUW5jtACLiJ3L3weIXzHjEg8FzlEZlkjuDI8SctZaXCXFNbCeSQFFcCP832vy/16AlJJbeWpmPDVpBdKLViIFAhZCIVM38GeYojbR06PA/ttP4jogYaNQODmw2fUIqhMRTfloAGUvNOGe3+fy/RcWfWhyVFkxxzSBUP+Yroz8hEalc1mZY/X9mfP7DwXWURlU06W4KoX1rod1I22kkUOxebD62uwg6j4bsn+DwTjh84mP+fjiWY/67O12PWkgYxKdVLI4STxpHxdXRG/FtKoREpHJfPwhul5kJat7fdhrfk9INfnhPDdPiWbuXmI8dR0H3a079enEBHN5VsTg6vNMUTEd2gbP4xP0/nf77xzQ882xS/VRznl4QUCEkIme3cyFs/SK4j9KojJbQi6eVHIO9q8z4TL98RMRCcidz+28uF+TtrVgcnVwwFR4qv+1ZeerzQyMgocUZZpNamNcOECqEROTMTj5Ko/cN0Kit1Tg+q/QU+iO7oTAbYhrYzSP+b+9qM6MTmwQNWlf/+SEhEN/M3FoOPvXrx3LLi6T/nk3KyTCvfWibuZ1OveRTL7WVFk31kv3q8rkKIRE5s40fmd8Ww2PNvkFyetEJ5gfA4Z2QuQ5aD7EcSPxeaX9Q8/7eKSqi4sweWKX7YJ3MeRxy95QXSCfPJmXvhKIc05+Uv7+8mftkYdGnFkelBVNCcwiP8vz7qQUVQiJyeseL4euHzHjQHVAvyW4eX5fSzfyw2LdWhZDUXmmB0XxA3b92aJi5/JXY4vRfP3r41Etth3eaIin3Zzh+FA5sMrdTOCAu9dTZpJaDra24VCEkIqcqyoOv/mD+kYtNggGTbCfyfU26mcNo1SckteVylTdK++LihOhEaJoITXud+rXjxebS2imzSSduxflmtil3D+xaWP68a/6uQkhEfIDLBWv/DvMeMtPeACOmQ2Q9u7n8QZMe5qNWjkltHdwCx46YS9Ipp7l05cvCIqBhG3P7b263ac6uUBydGFvsP/S7tXEvvvgiLVu2JCoqivT0dJYtW3bGx86aNYtzzz2XxMREEhMTGTFixFkfLxLUfl4Br42Af//WFEEN2sCv34ee19pO5h9Kf2Ad3GaWNYvUVGl/ULM+gbUhosMBsY0grS90uxKG/A5G/xVu/FyFUFW99957TJkyhenTp7Nq1Sq6d+/OyJEjycrKOu3j58+fz5gxY/j2229ZvHgxaWlpXHDBBezZs6eOk4v4sNx98K9b4NXhpjE6oj6c/wj8dok2B6yO+slQLwVww/4NttOIP/Ply2IByOF2+88pgenp6fTt25cXXjDnHblcLtLS0rj99tuZOrXyFS1Op5PExEReeOEFrr/++iq9Zm5uLvHx8eTk5BAXp104JYAcL4LFL8J/njLX7QF6XAvDHzA/1KX63rkStn0FF/8Z+k2wnUb81bNdzVYM130Ibc6zncZvVfXnt9/MuRUXF7Ny5UqmTZtWdl9ISAgjRoxg8eLFVfoehYWFlJSU0KDBmff4KCoqoqioqOzz3NzcmocW8UVuN2z5HL68v3zH2WZ94aInoGlvu9n8XZPuphBSn5DUVM4eUwQ5Qsz/l+J1flMIHTx4EKfTSXJyxd9Uk5OT2bx5c5W+x7333ktqaiojRow442NmzJjBQw89VKusIj4razPMmQo/fms+r98ERjwEXa8Mmu30vaq0T0iFkNRUxonLYildIbK+3SxBwm8Kodp6/PHHmT17NvPnzycq6sybOU2bNo0pU6aUfZ6bm0taWlpdRBTxnqOHYf4TsGwmuJ1m+/yBt8PgKVoR5kmlm9NlbTLLiMMi7OYR/1PWH2Rh/6Ag5TeFUKNGjQgNDWX//v0V7t+/fz8pKSlnfe6f//xnHn/8cb7++mu6dTv7UsTIyEgiIyNrnVfEJ7icsOot+OaPZtkqmINTL/ij2fFVPCuhBUTFmxPBD2wqP4NMpKpO3lFa6oTfzIVHRETQu3dv5s2bV3afy+Vi3rx5DBhw5sr5ySef5JFHHmHOnDn06dOnLqKK+IadC2HmEPj0LlMENe4A130E17yjIshbHI6TLo9pY0WppmO55SsO01QI1RW/mRECmDJlCuPGjaNPnz7069ePZ599loKCAm688UYArr/+epo2bcqMGTMAeOKJJ3jggQd49913admyJZmZmQDUq1ePevV0OUAC1JEMc1Dqhg/N51HxMOx+6DM+sPYk8VVNusPO/2iHaam+n5eD22WOnLC0y3Iw8qt/Fa+++moOHDjAAw88QGZmJj169GDOnDllDdS7d+8m5KSGz5deeoni4mJ+9atfVfg+06dP58EHH6zL6CLeV1wIi/4C3z9rzvpxhJgT44f9HmIb2k4XPEovh6lhWqpL/UFW+FUhBDBp0iQmTTr9uUfz58+v8PnOnTu9H0jENrfbzP7MfcCc8QPQYpBZDp/S1W62YFRaCGWuNz1aIaF284j/UH+QFX5XCInISTLXwRdTYdf35vP4NLjgEeg02vSrSN1reA6Ex0BJARzaAY3b2U4k/sBZYo65Ac0I1TEVQiL+qOAQfPtHWPmm6SkIi4bBk2HgHRARYztdcAsJheTOpt8j8wcVQlI1+34wl7SjG0Aj/TdTl1QIifgTZwksfw3mP2aWaAN0vhzOfxgStN+Vz2jS3RRC+9ZA119V+nCRsstiaemaza1jKoRE/MWOb82u0AdO7KSe0hUufAJaDrKbS06lJfRSXeoPskaFkIivy/7JnAu25TPzeUxDOO8P0Ot6NeL6qpNXjrnd+g1fzs7t1ooxi1QIifiqonxzMvziF8BZDI5Q6HczDL0XohNtp5OzSeoIIWFw7IhZyZfQ3HYi8WWHdkDhQQiNhNQettMEHRVCIr7G5YJ178Pc6ZBvNgGl9TC48HFI6mA3m1RNWKQphjLXmctjKoTkbEovizXtbf7bkTqlQkjEl+xZCV/caxptARJbwcjHoP1Furzib1K6nyiE1kLH/7GdRnxZ2WUx9QfZoEJIxBfk7Yd5D8Oat83n4bHwi3tgwET9huivmnSDNeioDalcWaO0+oNsUCEkYtPxYlj6Enz3JyjOM/d1HwPDp+usIX+nozakKvIPQPYOwAFpfW2nCUoqhERs2folzJl24h9BILUXXPSk/jEMFMldAAfk7YP8LKiXZDuR+KKME5fFkjppEYQlKoRE6trBbaYA2j7XfB6bBCMeNDNBJx0aLH4usp45buPQNtMw3XaE7UTii9QfZJ0KIZG6ciwHvnsSlr4MruMQEg4Dfgvn3gNRcbbTiTc06WYKocy1KoTk9NQfZJ0KIRFvczlhzTumGbrggLmv3YVmNVjDNnaziXc16Q7rP9AO03J6xQXlPWTN0+1mCWIqhES8afcS+OJ/y/+xa9QORs7Q7ECwKDtqQw3Tchp7VprZ4bimEK+zAm1RISTiDTl74OvpsO4f5vPIOBg61ewMHRpuN5vUndKVY4d/MpdGo+Lt5hHfcnJ/kPYJs0aFkIgnlRyDRc/D909DSSHgMGeCnfcHqNfYdjqpazENzG/6ORlmc8WWg20nEl+i/iCfoEJIxBPcbtj0CXx1PxzZbe5L6w8XPaGzg4JdSjdTCO37QYWQlHMeh4xlZqwVY1apEBKprf0bYM5U+GmB+TyuKZz/MHS5QtPdYi6PbflMfUJSUdYGKM43l82TOtlOE9RUCInUVGE2fPsYrHgN3C5zcvSgO2HwZIiItZ1OfEWTEw3TOmpDTlbaH5TWD0JC7WYJciqERGrieDG8doHZIwag46VwwR8hsYXdXOJ7ShumD2yBkqMQHm03j/iGsv4gXRazTYWQSE1s/LcpgmIawZVvQKtf2E4kvqp+E/PfSeFB2L8RmvW2nUhsc7tPWjGmRmnbtJ+/SHW53bDkRTPuf6uKIDk7h+Oky2PqExLMgoq8fWZ3+dRettMEPRVCItWVsRT2roawKOh9o+004g90Er2crHQ2KLUHRMRYjSIqhESqb8lfzcduV0FsI7tZxD+U7TCthmlB/UE+RoWQLbn7YP4TpulW/MeR3Wa/IID+v7WbRfxH6YzQ/g3gLLGbRewrWzGmQsgXqFnaBpcLXh8JR3ZBfDPoOdZ2IqmqZTPNUvnWwyCpo+004i8SW0FEfSjOg4NbIbmz7URiS2E2HNhkxpoR8gmaEbIhJAT63GTGi/5iCiPxfUX5sPJvZqzZIKmOkJDyhmldHgtupbtJN2yrS+s+QoWQLX1uNL8hHtgM2+faTiNVseZdKMqBhufAOTo9XqpJJ9ELqD/IB6kQsiUqHvrcYMYLn7MaRarA5YKlL5lx+q3mN3yR6ijtE9IO08FN+wf5HP1rblP6bWYfiV0L4ecVttPI2Wz7CrJ/NAVs9zG204g/OvnSmC6HB6eSY7B3lRlrRshnqBCyKb4pdL3SjDUr5NtKl8z3vgEi61mNIn6qUTtzHl1xHhz+yXYasWHvanAWQ2wSNGhtO42coELItoG3m4+bPoFDO+xmkdPbvwF++g4codB3gu004q9Cw8tXi6lPKDid3B/kcNjNImWqXAjl5uZW+SbVkNwJ2l4AuGHxC7bTyOmUzgZ1uhQS0uxmEf+mk+iDW8ZS81H9QT6lyvsIJSQk4KhiBet0OmscKCgNutP0oKx5F4beB/Ua204kpfIPwA//MGMtmZfaKjtqQ4VQ0HG5TmqUVn+QL6lyIfTtt9+WjXfu3MnUqVO54YYbGDDAVLaLFy/mrbfeYsaMGZ5PGehaDDIH7+1dZTbsO+9+24mk1Mo3wFkETXtDs76204i/SznpzDG3W5dHgsnBLXDsCITHlm+lID6hyoXQkCFDysYPP/wwTz/9NGPGlK+eufTSS+natSszZ85k3Lhxnk0Z6BwOMyv0j3GwfBYMngwRsbZTyfEiWP6qGff/rX5oSe0ldzK9ZoUHzenjcam2E0ldKe0PatYHQnWogy+pUbP04sWL6dOnzyn39+nTh2XLltU6VFDqOMpsw3/0MKx+23YaAdjwIeTvh/qp0Oky22kkEIRHQ+P2ZqyG6eCiy2I+q0aFUFpaGrNmzTrl/ldffZW0NDWT1khIKAycZMaLXwDncbt5gp3bDYtfNON+E8yKHxFP0En0wUk7SvusGs3PPfPMM1xxxRV88cUXpKenA7Bs2TK2bdvGBx984NGAQaXHWPj2MXPC+caPoOuvbCcKXrsWmZU9YdFm7yART2nSHX6YrRmhYJKzx/y77ghRr6EPqtGM0MUXX8y2bdu49NJLyc7OJjs7m1GjRrF161YuvvhiT2cMHuHR0O8WM170FzMrIXaULpnvfg3ENLCbRQKLltAHn4wTl8VSukJkfbtZ5BTVnhEqKSnhwgsv5OWXX+bRRx/1Rqbg1vc38P0z5rfFnxZA6yGVP0c8K/sn2PyZGaffajeLBJ6UruZjTgYUZqvQDgY6X8ynVXtGKDw8nB9+0G8yXhPbEHpdZ8Y6dsOOZbMAN7QZDkkdbKeRQBMVbxZGgC6PBQv1B/m0Gl0au/baa3nttdc8nUVKDZhoriXvmAeZ622nCS7HcmHV38x4gDZQFC/RSfTB41iuOaYHIE2FkC+qUbP08ePHef311/n666/p3bs3sbEV97x5+umnPRIuaCW2NMu1N3xoeoUun2k7UfBY8645FLNRezMjJOINTbqZBRGaEQp8Py8Ht8v8ux7XxHYaOY0aFULr16+nV69eAGzdurXC16p6DIdUYuAdphBa/wGc9wedcVUXXE5Y+pIZ979VGyiK96ToqI2gof4gn1ejQujk4zbES5r2gpbnws7/wJKX4MLHbCcKfFvnwOGdEJUA3a6xnUYCWenKsUPboSgfIuvZzSPeo/4gn1ejHiGpI4Mmm4+r3jI7Tot3LTkxG9TnRoiIsZtFAlu9JKjfBHDDfvUBBixnCfy8wow1I+SzanzgyYoVK3j//ffZvXs3xcXFFb72r3/9q9bBBDhnOCR1hqwNsOJ1OPdu24kC174fzOxbSBj0nWA7jQSDJt3NeWP71mq2IFDt+wGOH4XoBtCone00cgY1mhGaPXs2AwcOZNOmTXz44YeUlJSwYcMGvvnmG+Lj4z2dMXg5HDDoDjNe+gqUHLObJ5Atfdl87DQa4ptajSJBQkdtBL7Sy2Jp6eo59GE1KoQee+wxnnnmGT755BMiIiJ47rnn2Lx5M1dddRXNmzf3dMbg1uUKiGtqDv/84T3baQJT3n5Y9w8z7q8l81JHypbQa+VYwFJ/kF+oUSG0Y8cOLrnkEgAiIiIoKCjA4XBw1113MXOmlnp7VGh4+Q/nRc+Dy2U3TyBa8To4i6FZP2jW23YaCRalDdNZm+B4kd0s4nlut1aM+YkaFUKJiYnk5eUB0LRpU9avN81+R44cobCw0HPpxOg9DiLj4dA22PqF7TSBpeQYrDixOWj/2+xmkeASn2ZWKLqOm2JIAsuhHVB4EEIjIbWH7TRyFjUqhH7xi18wd+5cAK688kruvPNOJkyYwJgxYxg+XJvQeVxkfeh7kxkv/IvdLIFm/QdQcADimkHHS22nkWDicJRfHtPGioGn9LJY094QFmk3i5xVjVaNvfDCCxw7Zhp377//fsLDw1m0aBFXXHEFv//97z0aUE5IvxUWv2hOMd69FJqn207k/9zu8iXz/SZAaI0XUYrUTJNu8NN3OmojEJVdFlN/kK+r0b/8DRqUn5YcEhLC1KlTPRZIzqB+CnS7Glb/nzl2o/k7thP5v53fw/51EB5jLj+K1LUmPcxHzQgFnrJGafUH+boaXRq7/vrreeONN9ixY4en88jZDLzdfNz8GRzcZjdLIFjyV/Oxx68hOtFuFglOpUvo928wR7xIYMg/ANk7AAek9bWdRipRo0IoIiKCGTNm0LZtW9LS0rj22mt59dVX2bZNP5y9qnF7aH8x4DYryKTmDu2ALScaz9NvtZtFglfDNhAeCyWF5rgNCQwZJy6LJXXSL1l+oEaF0KuvvsrWrVvJyMjgySefpF69ejz11FN06NCBZs2aeTqjnGzgiQ0W1/7d7H8jNbNsJuCGthdAo7a200iwCgmFlC5mrMtjgUP9QX6lVmeNJSYm0rBhQxITE0lISCAsLIzGjRt7Kttpvfjii7Rs2ZKoqCjS09NZtmzZWR//j3/8gw4dOhAVFUXXrl35/PPPvZrP65r3N/vdOIth2Su20/inYzmw+m0z1pJ5sa1sh2kVQgFD/UF+pUaF0H333cfAgQNp2LAhU6dO5dixY0ydOpXMzExWr17t6Yxl3nvvPaZMmcL06dNZtWoV3bt3Z+TIkWRlZZ328YsWLWLMmDGMHz+e1atXM3r0aEaPHl2275FfOvnYjeWvmpOrpXpWvw3F+dC4I7QeZjuNBDstoQ8sxQXlf5da3esXHG63213dJ4WEhNC4cWPuuusuLr/8ctq1q5vD5NLT0+nbty8vvPACAC6Xi7S0NG6//fbTrly7+uqrKSgo4NNPPy27r3///vTo0YOXX365Sq+Zm5tLfHw8OTk5xMXFeeaN1JbLCS/0Nc14I2fAAB0LUWUuJ/ylBxzZDaP+otViYt++tfDKLyAqHu7dpTOp/N1PC+CtUeZopLs26O/Toqr+/K7RjNDq1au5//77WbZsGYMGDaJp06b8+te/ZubMmWzdurXGoc+muLiYlStXMmLEiLL7QkJCGDFiBIsXLz7tcxYvXlzh8QAjR4484+MBioqKyM3NrXDzOSGh5SvIlvwVnCV28/iTzZ+ZIii6AXS7ynYaETMzGRJuLtke2WU7jdTWyf1BKoL8Qo0Koe7du3PHHXfwr3/9iwMHDvD5558TERHBxIkT6dixo6czAnDw4EGcTifJyckV7k9OTiYzM/O0z8nMzKzW4wFmzJhBfHx82S0tLa324b2h+xiIbQw5GbDhQ9tp/EfpBop9boLwaLtZRADCIiDpxL+bOone/6k/yO/UqBByu92sWrWKp59+mksvvZRhw4bx9ttv07VrV+644w5PZ6xT06ZNIycnp+yWkZFhO9LphUdB+i1mvPAvZpdkObu9q2H3IggJg76/sZ1GpFzZSfQqhPya8zhknFjAoxVjfqPGO0vn5+fTvXt3hgwZwoQJEzj33HNJSEjwcLxyjRo1IjQ0lP37Ky4Z379/PykpKad9TkpKSrUeDxAZGUlkpJ+cC9NnPPznGbM78o5v4Byd83ZWS070hXW+HOKa2M0icrIm3c2u8WqY9m9ZG8xCjMg4s4eQ+IUazQi9/fbbHDp0iBUrVvDUU08xatQorxZBYDZx7N27N/PmzSu7z+VyMW/ePAYMOP0U5IABAyo8HmDu3LlnfLzfiWkAva4340U6jPWs8jLNAaugJfPie8qW0GtGyK+V9gel9TO9nOIXalQIXXLJJcTFxbF9+3a+/PJLjh49CphLZt40ZcoUZs2axVtvvcWmTZu47bbbKCgo4MYbbwTM0R/Tpk0re/ydd97JnDlzeOqpp9i8eTMPPvggK1asYNKkSV7NWacG/BYcofDjfNi7xnYa37X8VXCVmOv2TXvZTiNSUUoXwAH5mdoo1Z+V9Qfpspg/qVEhdOjQIYYPH067du24+OKL2bdvHwDjx4/n7rvv9mjAk1199dX8+c9/5oEHHqBHjx6sWbOGOXPmlDVE7969uywLwMCBA3n33XeZOXMm3bt355///CcfffQRXbp08VrGOpfQHLpcbsY6duP0So7CitfNWLNB4osiYst3OFefkH9yu09aMRYgVx2CRI0Kobvuuovw8HB2795NTExM2f1XX301c+bM8Vi405k0aRK7du2iqKiIpUuXkp5evmHV/PnzefPNNys8/sorr2TLli0UFRWxfv16Lr74Yq/ms6L02I0NH8JhLb89xbp/QOEhiG8O7S+xnUbk9Mo2VlxjNYbU0JHdkLfPbIWQqllnf1KjQuirr77iiSeeOOVcsbZt27Jrl34Q17km3cwOyW5n+YnqYrjd5Uvm02+G0BqtDxDxPvUJ+bfS2aDUHhARc9aHim+pUSFUUFBQYSaoVHZ2tv+suAo0pcdurPobFGbbzeJLfvoOsjaaE757Xmc7jciZaQm9f1N/kN+qUSF07rnn8re//a3sc4fDgcvl4sknn2TYMJ3dZEXrYZDSFUoKYflrttP4jtLZoJ5jITrBahSRs0rpaj4e3glHj9hMIjVRtmJMhZC/qVEh9Kc//YmZM2dy0UUXUVxczP/+7//SpUsXFixYwBNPPOHpjFIVDgcMvNOMl75sGoSD3cHtsHUO4ID0W22nETm7mAamjw0gc53dLFI9hdlwYJMZa0bI71S7ECopKeGOO+7gk08+YfDgwVx22WUUFBRw+eWXs3r1atq0aeONnFIVnUebf0gLD8Lav9tOY9/SExsotrsQGuq/S/EDTUr7hLSxol8p3U26YVuIbWQ3i1RbtTtHw8PD+eGHH0hMTOT+++/3RiapqdBws6/QnKlmKX2vccG7qdfRw7DmHTPWknnxF026w+ZP1Sfkb9Qf5NdqdGns2muv5bXX1Ifik3peB1EJkP2jOWk9WK36P9MvldQZWv3CdhqRqilbQq8ZIb+i/YP8Wo3WEh8/fpzXX3+dr7/+mt69exMbG1vh608//bRHwkkNRNYzB4r+58+w8DnoOMr0DwUT53FYNtOM+98WfO9f/FfpEvqDW6G4UMuw/UHJMdi7yow1I+SXalQIrV+/nl69zIZRW7durfA1h37o2Jd+i7k0tmeFmbJtMdB2orq1+RPIyYCYRtD1SttpRKqufgrEJkFBFuzfAGl9bSeSyuxdDc5i8/fWoLXtNFIDNSqEvv32W0/nEE+qlwQ9xsDKN82sULAVQqVL5vuOh/Aou1lEqsPhMA3T27+GzLUqhPzByf1BmgjwSzXqERI/MOB2wGGWj2dttp2m7vy8EjKWmm3u+4y3nUak+sr6hNQw7RcylpqP6g/yWyqEAlWjc6DDiXO1FgfRYaxLT8wGdf0V1E+2m0WkJlK0hN5vuFwnNUqrP8hfqRAKZINObLC49j3I3Wc3S13I3WsOngVtoCj+q3QvoayN4Cyxm0XO7uAWOHbEHOFTWsCK31EhFMjS+pnpWldJ+eaCgWz5q+A6Di0GmYMPRfxRYiuIjDcNuAeC6LK2PyrtD2rWRwc6+zEVQoFu4InDWFe8Dsdy7WbxpuJC8x5BGyiKf3M4ys8dU5+Qb9NlsYCgQijQtbsQGrWDolxY9ZbtNN7zw3tmN+mEFtD+YttpRGpHGyv6B+0oHRBUCAW6kBAYeLsZL/4rHC+2m8cb3O7yJfPptwbvsSISOEr7hHTUhu/K2QNHdoMjBJppmwN/pkIoGHS7GuolQ95eWP+B7TSet+Mb07QYUR96Xms7jUjtlc4IZa4zK5PE92ScuCyW0hUi69vNIrWiQigYhEWWr6Ja9BczgxJISmeDel4LUXF2s4h4QsO2EBYFxfnm3EDxPTpfLGCoEAoWfW6CiHpmSe72r22n8ZwDW2D7XMAB6TfbTiPiGaFhkNzZjDPVJ+ST1B8UMFQIBYvoBOh9gxkvfM5mEs8q3Rag/cU650cCixqmfdexHHMWHECaCiF/p0IomPS/DULCYOd/YM9K22lqrzAb1vzdjAf81m4WEU8r22FaDdM+5+fl4HZBYkuIa2I7jdSSCqFgEt8MuvzKjBf+xW4WT1j1Fhw/apoVWwyynUbEs06eEQq0vj5/t1vniwUSFULBZtCJDRY3fezfTZjOElg2y4z7/1anPkvgSeoEjlA4mg25e2ynkZOpPyigqBAKNsmd4ZwRZlp38Yu209Tcpo/ND4fYxtDlCttpRDwvPAqSOpqxLo/5DmcJ/LzCjDUjFBBUCAWj0sNYV78DBQftZqmpxX81H/v+xmwPIBKIdBK979n3g7kkH93A7Novfk+FUDBqeS406WH+Zy69vORPMpbDnhUQGgF9xttOI+I92mHa95ReFktL1yX5AKFCKBg5HOWzQstmmgNL/cmSE7NBXa+Ceo3tZhHxJi2h9z3qDwo4KoSCVcdLzQGlR7NhzTu201Rdzs+w8d9m3P9Wu1lEvC25i/mYu8d/L2MHErdbO0oHIBVCwSo0DAZMMuPFL4DLaTdPVS2bBW6nubyX0tV2GhHvioqDBm3MWLNC9h3aAYUHITQSUnvYTiMeokIomPUcaxr+Du80q7B8XXEBrHzTjPtrA0UJEuoT8h2ll8Wa9tYijQCiQiiYRcRCvwlmvPA539+0be3f4dgRSGwF7UbaTiNSN8r6hFQIWVd2WUz9QYFEhVCw63ezOeV672rY+b3tNGfmcsGSE+eK9b8NQkLt5hGpK1pC7zvKGqXVHxRIVAgFu9hG0GOsGfvyYaw75sGhbRAZBz1+bTuNSN0pnRHK3gHHcu1mCWb5B8zfAQ5I62s7jXiQCiGBARPBEQLb55afqOxrSpfM97oeIuvbzSJSl2IbQVxTM96/3m6WYJZx4rJYUieITrSbRTxKhZBAwzbQcZQZL3rebpbTydoEO74xxVppT5NIMNFJ9PapPyhgqRASo3SDxXX/gBwfO+BxyUvmY4dLILGl1SgiVmhjRfvUHxSwVAiJ0bQ3tBgMruPll6F8QcEh+OE9M9aSeQlWWkJvV3FBeRGqGaGAo0JIypXOCq18C44esRqlzMo34PgxczaafhOTYFU6I3RgM5Qcs5slGO1ZaX5JjGsKCWm204iHqRCScm3Ph8YdoTjPFCC2HS+G5a+acf/f6oBDCV5xTc3mp67jkLXRdprgo/6ggKZCSMo5HDDoDjNe8jIcL7KbZ+O/IW8f1EuGzr+0m0XEJodDl8dsUn9QQFMhJBV1+RXUT4X8TPjhfXs53G5Y8qIZ950AYRH2soj4AjVM2+E8DhnLzFgzQgFJhZBUFBZhdm4Gs5Te5bKTI2Op2e06NBL63Ggng4gv0RJ6O7I2QHG+2cw1qZPtNOIFKoTkVL1vMP/TH9wC2760k6F05Vq3q8yGciLBrkkP83H/ejNLIXWjtD8orZ+O9glQKoTkVFFx5bMwC/9S969/ZDds+sSMtWRexGjQGiLqmVWUh7bZThM8yvqDdFksUKkQktNLvw1CwmH3IshYXrevvWwmuF3QeigkaypaBICQEEjpasbqE6obbvdJK8bUKB2oVAjJ6cU1MZelABbV4WGsRfmw8m9mrNkgkYrUJ1S3juw2K1dDwiG1l+004iUqhOTMBt5uPm76FA5ur5vXXPMuFOVAgzZwzvl185oi/kJL6OtW6WxQag+IiLEaRbxHhZCcWVJHaDsScMPiF7z/ei4XLD1xrlj/28ylABEpV7aE/gdz2Ua8S/1BQUE/aeTsSo/dWPMu5Gd597W2fQXZP0JUPHQf493XEvFHjTtAaISZNT2803aawFe2YkyFUCBTISRn12KgOZDVWWSamL2pdMl8r3EQWc+7ryXij0LDy/eyUcO0dxVmw4FNZqwZoYCmQkjOzuEonxVaNss0M3vD/g3w03fgCIV+N3vnNUQCgfqE6kbpbtIN22ovswCnQkgq1+F/zB4mx47A6re98xpLTvQGdRyl051FzkZHbdQN9QcFDRVCUrmQUBgwyYwXv+j5XW3zD5Sfa6Yl8yJnl3JSw7R4j/YPChoqhKRqevwaYhpBzm7Y+JFnv/fKN0wPUmovs429iJxZcmdwhEBBFuRl2k4TmEqOwd5VZqwZoYCnQkiqJjwa0m8x44XPeW7p7vEiWP6qGff/relJEpEzi4iBRu3MWJfHvGPvanAWQ2ySaQuQgKZCSKqu728gPMY0af443zPfc8OHkL8f6jeBzqM98z1FAp12mPauk/uD9MtZwFMhJFUX0wB6XmfGizxwGKvbbXqOAPpNMEuDRaRyZQ3Ta6zGCFgZS81H9QcFBRVCUj0Dfmv6E3Z8U/vfRncvNrNLYVHQ+0bP5BMJBlpC7z0u10mN0uoPCgZ+UwhlZ2czduxY4uLiSEhIYPz48eTnn3lPm+zsbG6//Xbat29PdHQ0zZs354477iAnJ6cOUwegxJbQ+ZdmvOj52n2v0tmg7teY2SYRqZrSS2NHdpuN/8RzDm4xW4WEx5b/OUtA85tCaOzYsWzYsIG5c+fy6aefsmDBAm6++cwb7+3du5e9e/fy5z//mfXr1/Pmm28yZ84cxo8fX4epA9TAO8zH9R+Yf4hrIvsn2PyZGaff5plcIsEiOgESWphx5jqrUQJOaX9Qsz4QGmY3i9QJvyiENm3axJw5c3j11VdJT09n8ODBPP/888yePZu9e/ee9jldunThgw8+YNSoUbRp04bzzjuPRx99lE8++YTjxz28D06wSe0BrYaA21m+EWJ1LZsFuKHNeZDUwZPpRIJDaZ+QLo95li6LBR2/KIQWL15MQkICffr0KbtvxIgRhISEsHTp0ip/n5ycHOLi4ggLO3OVX1RURG5uboWbnMagE7NCK9+Co4er99xjubDqb2bcf6Jnc4kEi9I+IS2h9yztKB10/KIQyszMJCkpqcJ9YWFhNGjQgMzMqm0odvDgQR555JGzXk4DmDFjBvHx8WW3tDQd93BabYZDchcoKYDlr1XvuWveheI8sxdKm/O8k08k0GmHac/L2WMu9ztCoFlf22mkjlgthKZOnYrD4TjrbfPmzbV+ndzcXC655BI6derEgw8+eNbHTps2jZycnLJbRkZGrV8/IDkc5b1CS18xO7FWhcsJS09cTku/FUL8ohYX8T2ll8YOboXiArtZAkXGictiKV0hsr7dLFJnrHaC3X333dxwww1nfUzr1q1JSUkhKyurwv3Hjx8nOzublJSUsz4/Ly+PCy+8kPr16/Phhx8SHn72vWoiIyOJjIysUv6g1+VymPcw5P4MP8yG3jdU/pytc+DwTohKMKvFRKRm6idDvWSzIen+DTqexhN0vlhQsloINW7cmMaNG1f6uAEDBnDkyBFWrlxJ7969Afjmm29wuVykp6ef8Xm5ubmMHDmSyMhIPv74Y6KiojyWXTAbIA74LXx5n1lK3/P6ymd4Spure98AEbFejygS0Jp0h21fmT4hFUK1p/6goOQX1yU6duzIhRdeyIQJE1i2bBkLFy5k0qRJXHPNNaSmpgKwZ88eOnTowLJlywBTBF1wwQUUFBTw2muvkZubS2ZmJpmZmTidTptvJ7D0uh4i4+HQdtjy+dkfu+8H2PkfcISanaRFpHZS1DDtMcdyzMwaQJoKoWDiF4UQwDvvvEOHDh0YPnw4F198MYMHD2bmzJllXy8pKWHLli0UFhYCsGrVKpYuXcq6des455xzaNKkSdlNfT8eFFkf+p7Ym2nhc2d/7NKXzcfOoyG+mVdjiQQFLaH3nJ+Xg9tlNo2Na2I7jdQhv9ktqkGDBrz77rtn/HrLli1xn3Qi+tChQyt8Ll6UfgssfgF+XmausZ9uWjk/C9b9w4z7/7Zu84kEqtIl9Ps3wvFiCIuwm8ef7db5YsHKb2aExIfVTylvfF54hsNYl78GzmKzJLVZn9M/RkSqJ6EFRMWDqwQO1H6FbVBTf1DQUiEknjHgdvNxy2dwYGvFr5UcgxUn9hrqr+M0RDzG4VCfkCc4S+DnFWasGaGgo0JIPKNxO2h/iRkv/q/DWNd/AAUHIK4pdLy07rOJBDL1CdXevh/g+FGIbmA2epWgokJIPKf02I21syHvxI7fbnf5kvl+E8ySexHxnNJCSDNCNVd6WSwt3cyySVBRISSe07y/+YfEWWx2mwbY+T3sXwfhMdBrnN18IoGo9NJY5nqzc7tUn/qDgpoKIfGs0mM3lr8GRXmw5K/m8+5jIKaBvVwigapRWwiLNuf+HdphO43/cbu1o3SQUyEkntX+Ymh4DhTlwNcPwpYvzP3pt1qNJRKwQkIhpYsZq0+o+g7tgMKDEBoJqT1spxELVAiJZ4WEwMATK8iWvwq44ZzzTTO1iHiHVo7VXOllsaa9IUznTAYjFULied2ugdik8s+1ZF7Eu9QwXXNll8XUHxSsVAiJ54VHmd2mARp3gDbn2c0jEuhKd5jO/MH0vEjVlTVKqz8oWPnNERviZwbeAaER0PYCLUcV8bakThASBkcPQ04GJDS3ncg/5GdB9g7AAWl9bacRSzQjJN4RFmH2FUrqYDuJSOALi4TGHc14nxqmqyzjxPliSZ0gOtFuFrFGhZCISCBQn1D1qT9IUCEkIhIYTu4TkqpRf5CgQkhEJDCULaFXIVQlxQXls2eaEQpqKoRERAJBShfAAXl7If+A7TS+b89KcB03h0EnpNlOIxapEBIRCQSR9aFhGzPOVJ9QpdQfJCeoEBIRCRRqmK469QfJCSqEREQChfqEqsZ5HDKWmbFmhIKeCiERkUChGaGqydoAxfkQGWf2EJKgpkJIRCRQlBZCh3+CYzl2s/iy0v6gtH4QEmo3i1inQkhEJFDENIC4Zmacud5uFl9W1h+ky2KiQkhEJLDo8tjZud0nrRhTo7SoEBIRCSzaYfrsjuyGvH0QEg6pvWynER+gQkhEJJBoRujsSmeDUntARIzVKOIbVAiJiASS0iX0B7ZAyVG7WXyR+oPkv6gQEhEJJHGpENMI3E7Yv9F2Gt9TtmJMhZAYKoRERAKJw3FSn5Auj1VQmA0HNpmxZoTkBBVCIiKBRjtMn17pbtIN20JsI7tZxGeoEBIRCTRqmD499QfJaagQEhEJNKWF0P4N4Cyxm8WXaP8gOQ0VQiIigSaxFUTUB2cRHNxqO41vKDkGe1eZsWaE5CQqhEREAk1ICKR0NWP1CRl7V4OzGGKToEFr22nEh6gQEhEJROoTqujk/iCHw24W8SkqhEREApGO2qgoY6n5qP4g+S8qhEREAlHpEvrMdeBy2c1im8t1UqO0+oOkIhVCIiKBqHF7CI2EolxY9Bdz6nqwOrgFjh2B8NjyAlHkBBVCIiKBKDQc0m8246+nwz9vhKJ8u5lsKe0PatYHQsPsZhGfo0JIRCRQnf8IXPQnCAmDDR/CrPPg4DbbqeqeLovJWagQEhEJVA6HmRW64XOo38RcIpo5DDZ+bDtZ3dKO0nIWKoRERAJd83S4+TtoMQiK8+D962DudHAet53M+3L2wJHd4AiBZn1tpxEfpEJIRCQY1E+G6/8NAyaZzxc+C2//EvIPWI3ldRknLouldIXI+naziE9SISQiEixCw2Hko/CrN8wKqp8WwMwh8PMK28m8R+eLSSVUCImIBJsul8OEb6DhOZC7B964CFa8HphL7NUfJJVQISQiEoySOsCEb6HD/5gzuD69C/49EUqO2k7mOcdyYP8GM05TISSnp0JIRCRYRcXB1W/DiIdMM/Gad+C1C+DwTtvJPOPn5eB2QWJLiGtiO434KBVCIiLBzOGAwZPhuo8gpqE5m+yVIbDta9vJam+3zheTyqkQEhERaD0EblkATXub4yje+RV896R/n1Om/iCpAhVCIiJixDeDG7+A3jcCbvj2Ufj7NXD0sO1k1ecsKV8NpxkhOQsVQiIiUi4sEkY9C5e9aA5t3fYlzBxqTrH3J/t+gONHIboBNGpnO434MBVCIiJyqp7XwvivIKG5aZ5+9XxY+57tVFVXelksLd30QYmcgQohERE5vdQe5miONsPN7MqHN8Nn98DxYtvJKqf+IKkiFUIiInJmMQ1g7D9gyL3m8+Wz4M1LIHev3Vxn43ZrR2mpMhVCIiJydiGhMOw+GPMeRMXDz8vglV/Azu9tJzu9Qzug8KDpcUrtYTuN+DgVQiIiUjXtL4Sb50NyFyg4AG9dCote8L2jOUovizXtbZq/Rc5ChZCIiFRdg9Ywfi50uxrcTvjqfvjnjVCUbztZubLLYuoPksqpEBIRkeqJiIFfvgIX/xlCwmDDhzDrPDiw1XYyo6xRWv1BUjkVQiIiUn0OB/SbADd8DvWbwMEtphja+LHdXPlZkL0DcEBaX7tZxC+oEBIRkZprnm6W2LcYBMV58P51MPcBcB63kyfjxPliSZ0gOtFOBvErflMIZWdnM3bsWOLi4khISGD8+PHk51ftmrTb7eaiiy7C4XDw0UcfeTeoiEiwqZ8M1/8bBkwyny98Dt7+JeQfqPss6g+SavKbQmjs2LFs2LCBuXPn8umnn7JgwQJuvvnmKj332WefxaGdRUVEvCc0HEY+Cle+CeGx8NMCmDmk/LyvuqL+IKkmvyiENm3axJw5c3j11VdJT09n8ODBPP/888yePZu9e8++qdeaNWt46qmneP311+sorYhIEOv8S5jwDTRsC7l74I2LYMXrdbPEvrgA9q01Y80ISRX5RSG0ePFiEhIS6NOnT9l9I0aMICQkhKVLl57xeYWFhfz617/mxRdfJCUlpUqvVVRURG5uboWbiIhUQ1IHUwx1HAXOYvj0Lvj3RCg56t3X3bMSXMchrikkpHn3tSRg+EUhlJmZSVJSUoX7wsLCaNCgAZmZmWd83l133cXAgQO57LLLqvxaM2bMID4+vuyWlqb/mUREqi0qDq76PxjxEDhCYM078NoF5gBXb1F/kNSA1UJo6tSpOByOs942b95co+/98ccf88033/Dss89W63nTpk0jJyen7JaRkVGj1xcRCXoOBwyeDNd9BDENIfMHeGUIbJvrnddTf5DUQJjNF7/77ru54YYbzvqY1q1bk5KSQlZWVoX7jx8/TnZ29hkveX3zzTfs2LGDhISECvdfccUVnHvuucyfP/+0z4uMjCQyUluyi4h4TOshcMsCeP96c/nqnSth6DT4xe8gxEO/jzuPQ8YyM9aMkFSDw+32tUNiTrVp0yY6derEihUr6N27NwBfffUVF154IT///DOpqamnPCczM5ODBw9WuK9r164899xzjBo1ilatWlXptXNzc4mPjycnJ4e4uLjavxkRkWB1vAjmTDXN0wBtR8Llr3hmv599a81BsJFxcO9Oc1CsBLWq/vz2ix6hjh07cuGFFzJhwgSWLVvGwoULmTRpEtdcc01ZEbRnzx46dOjAsmXmN4KUlBS6dOlS4QbQvHnzKhdBIiLiQWGR8D/PwGV/hbAo2PYlzBwKmetq/71L+4PS+qkIkmrxi0II4J133qFDhw4MHz6ciy++mMGDBzNz5syyr5eUlLBlyxYKCwstphQRkUr1HAvjv4KE5qZ5+tXzYe17tfueZf1Buiwm1eMXl8Zs0qUxEREvKcyGf02A7V+bz/tOgJGPQVhE9b6P2w1Pd4S8fXDDZ9BysOezit8JqEtjIiISgGIawK/fhyH3ms+Xz4I3L4Hcs2+Ue4oju00RFBIOqb08n1MCmgohERGxJyQUht0HY96DqHj4eZlpev7pP1X/HqX9Qak9ICLGKzElcKkQEhER+9pfCDfPh+SuUHAA/nYZLHq+akdzqD9IakGFkIiI+IYGrU0TdbdrwO2Er34P/7gBivLO/ryyFWMqhKT6VAiJiIjviIiBX74MF//Z9Pxs/AhmDYcDW0//+MJsOLDJjDUjJDWgQkhERHyLwwH9JsCNn0P9JnBwC8w6DzZ+fOpjS3eTbtgWYhvVbU4JCCqERETEN6X1M0dztBgMxXnw/nUw9wFznEYp9QdJLakQEhER31UvCa7/NwyYZD5f+By8/UvIP2A+LztxXgetSs2oEBIREd8WGgYjH4Ur34TwWPhpAcwcAju/h72rzGM0IyQ1pEJIRET8Q+dfwoRvTD9Q7h6z+aKzGGKTzIozkRpQISQiIv4jqYMphjpeWn5f8/6mwVqkBlQIiYiIf4mKg6v+Buc/DPFp0Ot624nEj4XZDiAiIlJtDgcMutPcRGpBM0IiIiIStFQIiYiISNBSISQiIiJBS4WQiIiIBC0VQiIiIhK0VAiJiIhI0FIhJCIiIkFLhZCIiIgELRVCIiIiErRUCImIiEjQUiEkIiIiQUuFkIiIiAQtFUIiIiIStFQIiYiISNAKsx3A17ndbgByc3MtJxEREZGqKv25Xfpz/ExUCFUiLy8PgLS0NMtJREREpLry8vKIj48/49cd7spKpSDncrnYu3cv9evXx+FweOz75ubmkpaWRkZGBnFxcR77vr4k0N+j3p//C/T3GOjvDwL/Per91Zzb7SYvL4/U1FRCQs7cCaQZoUqEhITQrFkzr33/uLi4gPyP+2SB/h71/vxfoL/HQH9/EPjvUe+vZs42E1RKzdIiIiIStFQIiYiISNBSIWRJZGQk06dPJzIy0nYUrwn096j35/8C/T0G+vuDwH+Pen/ep2ZpERERCVqaERIREZGgpUJIREREgpYKIREREQlaKoREREQkaKkQsuTFF1+kZcuWREVFkZ6ezrJly2xH8pgFCxYwatQoUlNTcTgcfPTRR7YjedSMGTPo27cv9evXJykpidGjR7NlyxbbsTzmpZdeolu3bmUbnA0YMIAvvvjCdiyvefzxx3E4HEyePNl2FI958MEHcTgcFW4dOnSwHcuj9uzZw7XXXkvDhg2Jjo6ma9eurFixwnYsj2nZsuUpf4cOh4OJEyfajuYRTqeTP/zhD7Rq1Yro6GjatGnDI488Uum5YN6gQsiC9957jylTpjB9+nRWrVpF9+7dGTlyJFlZWbajeURBQQHdu3fnxRdftB3FK7777jsmTpzIkiVLmDt3LiUlJVxwwQUUFBTYjuYRzZo14/HHH2flypWsWLGC8847j8suu4wNGzbYjuZxy5cv55VXXqFbt262o3hc586d2bdvX9nt+++/tx3JYw4fPsygQYMIDw/niy++YOPGjTz11FMkJibajuYxy5cvr/D3N3fuXACuvPJKy8k844knnuCll17ihRdeYNOmTTzxxBM8+eSTPP/883Ufxi11rl+/fu6JEyeWfe50Ot2pqanuGTNmWEzlHYD7ww8/tB3Dq7KystyA+7vvvrMdxWsSExPdr776qu0YHpWXl+du27ate+7cue4hQ4a477zzTtuRPGb69Onu7t27247hNffee6978ODBtmPUqTvvvNPdpk0bt8vlsh3FIy655BL3TTfdVOG+yy+/3D127Ng6z6IZoTpWXFzMypUrGTFiRNl9ISEhjBgxgsWLF1tMJjWVk5MDQIMGDSwn8Tyn08ns2bMpKChgwIABtuN41MSJE7nkkksq/L8YSLZt20ZqaiqtW7dm7Nix7N6923Ykj/n444/p06cPV155JUlJSfTs2ZNZs2bZjuU1xcXFvP3229x0000ePfzbpoEDBzJv3jy2bt0KwNq1a/n++++56KKL6jyLDl2tYwcPHsTpdJKcnFzh/uTkZDZv3mwpldSUy+Vi8uTJDBo0iC5dutiO4zHr1q1jwIABHDt2jHr16vHhhx/SqVMn27E8Zvbs2axatYrly5fbjuIV6enpvPnmm7Rv3559+/bx0EMPce6557J+/Xrq169vO16t/fjjj7z00ktMmTKF++67j+XLl3PHHXcQERHBuHHjbMfzuI8++ogjR45www032I7iMVOnTiU3N5cOHToQGhqK0+nk0UcfZezYsXWeRYWQSC1MnDiR9evXB1T/BUD79u1Zs2YNOTk5/POf/2TcuHF89913AVEMZWRkcOeddzJ37lyioqJsx/GKk3+r7tatG+np6bRo0YL333+f8ePHW0zmGS6Xiz59+vDYY48B0LNnT9avX8/LL78ckIXQa6+9xkUXXURqaqrtKB7z/vvv88477/Duu+/SuXNn1qxZw+TJk0lNTa3zv0MVQnWsUaNGhIaGsn///gr379+/n5SUFEuppCYmTZrEp59+yoIFC2jWrJntOB4VERHBOeecA0Dv3r1Zvnw5zz33HK+88orlZLW3cuVKsrKy6NWrV9l9TqeTBQsW8MILL1BUVERoaKjFhJ6XkJBAu3bt2L59u+0oHtGkSZNTivKOHTvywQcfWErkPbt27eLrr7/mX//6l+0oHvW73/2OqVOncs011wDQtWtXdu3axYwZM+q8EFKPUB2LiIigd+/ezJs3r+w+l8vFvHnzAq4HI1C53W4mTZrEhx9+yDfffEOrVq1sR/I6l8tFUVGR7RgeMXz4cNatW8eaNWvKbn369GHs2LGsWbMm4IoggPz8fHbs2EGTJk1sR/GIQYMGnbJlxdatW2nRooWlRN7zxhtvkJSUxCWXXGI7ikcVFhYSElKxBAkNDcXlctV5Fs0IWTBlyhTGjRtHnz596NevH88++ywFBQXceOONtqN5RH5+foXfPH/66SfWrFlDgwYNaN68ucVknjFx4kTeffdd/v3vf1O/fn0yMzMBiI+PJzo62nK62ps2bRoXXXQRzZs3Jy8vj3fffZf58+fz5Zdf2o7mEfXr1z+lnys2NpaGDRsGTJ/XPffcw6hRo2jRogV79+5l+vTphIaGMmbMGNvRPOKuu+5i4MCBPPbYY1x11VUsW7aMmTNnMnPmTNvRPMrlcvHGG28wbtw4wsIC68f1qFGjePTRR2nevDmdO3dm9erVPP3009x00011H6bO16mJ2+12u59//nl38+bN3REREe5+/fq5lyxZYjuSx3z77bdu4JTbuHHjbEfziNO9N8D9xhtv2I7mETfddJO7RYsW7oiICHfjxo3dw4cPd3/11Ve2Y3lVoC2fv/rqq91NmjRxR0REuJs2beq++uqr3du3b7cdy6M++eQTd5cuXdyRkZHuDh06uGfOnGk7ksd9+eWXbsC9ZcsW21E8Ljc3133nnXe6mzdv7o6KinK3bt3aff/997uLiorqPIvD7bawjaOIiIiID1CPkIiIiAQtFUIiIiIStFQIiYiISNBSISQiIiJBS4WQiIiIBC0VQiIiIhK0VAiJiIhI0FIhJCIiIkFLhZCI+KT58+fjcDg4cuSI7SgiEsBUCImITxg6dCiTJ08u+3zgwIHs27eP+Ph4a5lUjIkEvsA6xU1EAkZERAQpKSm2Y4hIgNOMkIhYd8MNN/Ddd9/x3HPP4XA4cDgcvPnmmxVmY958800SEhL49NNPad++PTExMfzqV7+isLCQt956i5YtW5KYmMgdd9yB0+ks+95FRUXcc889NG3alNjYWNLT05k/f37Z13ft2sWoUaNITEwkNjaWzp078/nnn7Nz506GDRsGQGJiIg6HgxtuuAEwp4LPmDGDVq1aER0dTffu3fnnP/9Z9j1LZ5I+++wzunXrRlRUFP3792f9+vVe/7MUkerRjJCIWPfcc8+xdetWunTpwsMPPwzAhg0bTnlcYWEhf/nLX5g9ezZ5eXlcfvnl/PKXvyQhIYHPP/+cH3/8kSuuuIJBgwZx9dVXAzBp0iQ2btzI7NmzSU1N5cMPP+TCCy9k3bp1tG3blokTJ1JcXMyCBQuIjY1l48aN1KtXj7S0ND744AOuuOIKtmzZQlxcHNHR0QDMmDGDt99+m5dffpm2bduyYMECrr32Who3bsyQIUPK8v7ud7/jueeeIyUlhfvuu49Ro0axdetWwsPD6+BPVUSqpM7PuxcROY0hQ4a477zzzrLPv/32WzfgPnz4sNvtdrvfeOMNN+Devn172WNuueUWd0xMjDsvL6/svpEjR7pvueUWt9vtdu/atcsdGhrq3rNnT4XXGj58uHvatGlut9vt7tq1q/vBBx88bab/zuB2u93Hjh1zx8TEuBctWlThsePHj3ePGTOmwvNmz55d9vVDhw65o6Oj3e+9914V/0REpC5oRkhE/EZMTAxt2rQp+zw5OZmWLVtSr169CvdlZWUBsG7dOpxOJ+3atavwfYqKimjYsCEAd9xxB7fddhtfffUVI0aM4IorrqBbt25nzLB9+3YKCws5//zzK9xfXFxMz549K9w3YMCAsnGDBg1o3749mzZtqua7FhFvUiEkIn7jvy8pORyO097ncrkAyM/PJzQ0lJUrVxIaGlrhcaXF029+8xtGjhzJZ599xldffcWMGTN46qmnuP3220+bIT8/H4DPPvuMpk2bVvhaZGRkzd+ciFihQkhEfEJERESFJmdP6NmzJ06nk6ysLM4999wzPi4tLY1bb72VW2+9lWnTpjFr1ixuv/12IiIiACrk6tSpE5GRkezevbtCP9DpLFmyhObNmwNw+PBhtm7dSseOHT3wzkTEU1QIiYhPaNmyJUuXLmXnzp3Uq1evbFanNtq1a8fYsWO5/vrreeqpp+jZsycHDhxg3rx5dOvWjUsuuYTJkydz0UUX0a5dOw4fPsy3335bVqy0aNECh8PBp59+ysUXX0x0dDT169fnnnvu4a677sLlcjF48GBycnJYuHAhcXFxjBs3ruz1H374YRo2bEhycjL3338/jRo1YvTo0bV+XyLiOVo+LyI+4Z577iE0NJROnTrRuHFjdu/e7ZHv+8Ybb3D99ddz99130759e0aPHs3y5cvLZmqcTicTJ06kY8eOXHjhhbRr146//vWvADRt2pSHHnqIqVOnkpyczKRJkwB45JFH+MMf/sCMGTPKnvfZZ5/RqlWrCq/9+OOPc+edd9K7d28yMzP55JNPymaZRMQ3ONxut9t2CBGRQDJ//nyGDRvG4cOHSUhIsB1HRM5CM0IiIiIStFQIiYiISNDSpTEREREJWpoREhERkaClQkhERESClgohERERCVoqhERERCRoqRASERGRoKVCSERERIKWCiEREREJWiqEREREJGj9P8YqygSSKeaxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's visualize the transition of reward\n",
    "obs, info = env.reset()\n",
    "done = False\n",
    "reward_list = []\n",
    "\n",
    "while not done:\n",
    "    action = agent.sample_action_online(obs)\n",
    "\n",
    "    obs, reward, done, truncated, info = env.step(action)\n",
    "    reward_list.append(reward)\n",
    "\n",
    "# plot\n",
    "fig = plt.figure()\n",
    "ax1 = fig.add_subplot(111)\n",
    "ax1.plot(reward_list[:-1], label='reward', color='tab:orange')\n",
    "ax1.set_xlabel('timestep')\n",
    "ax1.set_ylabel('reward')\n",
    "ax1.legend(loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more about the environmental configuration , please refer to [examples/quickstart/rec/rec_synthetic_customize_env.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_customize_env.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Discrete Action Case\n",
    "Here, we present how to collect logged data by a behavior policy in the case of discrete action.\n",
    "\n",
    "The procedure requires two steps:\n",
    "\n",
    "1. Learn a base deterministic policy\n",
    "2. Convert the deterministic policy into a stochastic policy.\n",
    "\n",
    "Below, we first learn a deterministic policy using [d3rlpy](https://github.com/takuseno/d3rlpy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# standardized environment for discrete action\n",
    "env = RECEnv(random_state=random_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learn a base deterministic policy for data collection\n",
    "from d3rlpy.algos import DoubleDQNConfig\n",
    "from d3rlpy.models.encoders import VectorEncoderFactory\n",
    "from d3rlpy.models.q_functions import MeanQFunctionFactory\n",
    "from d3rlpy.dataset import create_fifo_replay_buffer\n",
    "from d3rlpy.algos import LinearDecayEpsilonGreedy\n",
    "\n",
    "# model\n",
    "ddqn = DoubleDQNConfig(\n",
    "    encoder_factory=VectorEncoderFactory(hidden_units=[30, 30]),\n",
    "    q_func_factory=MeanQFunctionFactory(),\n",
    "    target_update_interval=100,\n",
    ").create(device=device)\n",
    "\n",
    "# replay buffer\n",
    "buffer = create_fifo_replay_buffer(\n",
    "    limit=10000,\n",
    "    env=env,\n",
    ")\n",
    "\n",
    "# explorers\n",
    "explorer = LinearDecayEpsilonGreedy(\n",
    "    start_epsilon=1.0,\n",
    "    end_epsilon=0.1,\n",
    "    duration=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# start training\n",
    "# skip if there is a pre-trained model\n",
    "ddqn.fit_online(\n",
    "    env,\n",
    "    buffer,\n",
    "    explorer=explorer,\n",
    "    eval_env=env,\n",
    "    n_steps=100000,\n",
    "    n_steps_per_epoch=1000,\n",
    "    update_start_step=1000,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save model\n",
    "ddqn.save_model(\"d3rlpy_logs/ddqn.pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-07-22 16:18:55 [warning  ] Parameters will be reinitialized.\n"
     ]
    }
   ],
   "source": [
    "# reload model\n",
    "ddqn.build_with_env(env)\n",
    "ddqn.load_model(\"d3rlpy_logs/ddqn.pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Epsilon-Greedy behavior policy\n",
    "\n",
    "Let's now convert the deterministic policy (i.e., ddqn policy) into a stochastic behavior policy.\n",
    "\n",
    "We use epsilon-greedy policy to collect logged data using `EpsilonGreedyHead`, where the behavior policy greedily takes an action chosen by the deterministic policy with probability $1 - \\epsilon$ and takes an action randomly with probability $\\epsilon$.\n",
    "\n",
    "Note that, `SyntheticDataset` has the following arguments:\n",
    "- `env`: REC environment for RL defined in the previous section.\n",
    "- `max_episode_steps`: Maximum number of timesteps in an episode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert the base ddqn policy into a stochastic data collection policy\n",
    "from scope_rl.policy import EpsilonGreedyHead\n",
    "\n",
    "behavior_policy = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.3,  # probability of taking random action\n",
    "    name=\"ddqn_epsilon_0.3\",\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect data\n",
    "dataset = SyntheticDataset(\n",
    "    env=env,\n",
    "    max_episode_steps=env.step_per_episode,\n",
    ")\n",
    "logged_dataset = dataset.obtain_episodes(\n",
    "    behavior_policies=behavior_policy,\n",
    "    n_trajectories=10000, \n",
    "    obtain_info=False,\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'size': 100000,\n",
       " 'n_trajectories': 10000,\n",
       " 'step_per_trajectory': 10,\n",
       " 'action_type': 'discrete',\n",
       " 'n_actions': 5,\n",
       " 'action_dim': None,\n",
       " 'action_meaning': None,\n",
       " 'action_keys': None,\n",
       " 'state_dim': 5,\n",
       " 'state_keys': None,\n",
       " 'state': array([[ 0.04479014,  0.11492637, -0.05131316, -0.0280654 ,  0.0347718 ],\n",
       "        [ 0.25961711,  0.85642393, -0.29858954, -0.12037568,  0.30901616],\n",
       "        [ 0.30441728,  0.84080503, -0.18078032, -0.14317481,  0.38366192],\n",
       "        ...,\n",
       "        [ 0.0653899 , -0.60680683,  0.19942688, -0.47770232, -0.59961575],\n",
       "        [ 0.08098473, -0.55363545,  0.25202619, -0.4167451 , -0.67062327],\n",
       "        [ 0.09878847, -0.61061154,  0.1608414 , -0.40941901, -0.65107642]]),\n",
       " 'action': array([3, 1, 3, ..., 2, 3, 2]),\n",
       " 'reward': array([ 0.0457314 , -0.31700356,  0.40663493, ..., -0.13823452,\n",
       "        -0.50789186, -0.18262118]),\n",
       " 'done': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'terminal': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'info': {},\n",
       " 'pscore': array([0.76, 0.06, 0.76, ..., 0.06, 0.76, 0.06]),\n",
       " 'behavior_policy': 'ddqn_epsilon_0.3',\n",
       " 'dataset_id': 0}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logged_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Softmax behavior policy\n",
    "We can also use `SoftmaxHead` to derive a stochastic behavior policy.\n",
    "\n",
    "This algorithm uses Q function of the original algorithm, which estimates the value of a given context and action pair (i.e., $(s, a)$) as $Q(s, a)$. \\\n",
    "Specifically, the behavior policy chooses actions stochastically as $\\pi(a \\mid s) = \\frac{\\exp(Q(s, a) / \\tau)}{\\sum_{a' \\in A} \\exp(Q(s, a') / \\tau)}$, where $A$ indicates the set discrete actions and $\\tau$ is an inverse temperature parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert base ddqn policy into a stochastic data collection policy\n",
    "from scope_rl.policy import SoftmaxHead\n",
    "\n",
    "behavior_policy = SoftmaxHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    tau=1.0,  # temperature parameter\n",
    "    name=\"ddqn_softmax_tau_1.0\",\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collect data\n",
    "dataset = SyntheticDataset(\n",
    "    env=env,\n",
    "    max_episode_steps=env.step_per_episode,\n",
    ")\n",
    "logged_dataset = dataset.obtain_episodes(\n",
    "    behavior_policies=behavior_policy, \n",
    "    n_trajectories=10000, \n",
    "    obtain_info=False,\n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'size': 100000,\n",
       " 'n_trajectories': 10000,\n",
       " 'step_per_trajectory': 10,\n",
       " 'action_type': 'discrete',\n",
       " 'n_actions': 5,\n",
       " 'action_dim': None,\n",
       " 'action_meaning': None,\n",
       " 'action_keys': None,\n",
       " 'state_dim': 5,\n",
       " 'state_keys': None,\n",
       " 'state': array([[ 0.04479014,  0.11492637, -0.05131316, -0.0280654 ,  0.0347718 ],\n",
       "        [ 0.25961711,  0.85642393, -0.29858954, -0.12037568,  0.30901616],\n",
       "        [ 0.19436985,  0.88078279, -0.22501215, -0.03685871,  0.36667834],\n",
       "        ...,\n",
       "        [ 0.09529246, -0.70234252,  0.10841566, -0.41846472, -0.55746538],\n",
       "        [ 0.10943013, -0.71880701,  0.04250762, -0.40402341, -0.5534435 ],\n",
       "        [ 0.11996917, -0.72904206, -0.00852912, -0.39147998, -0.54843026]]),\n",
       " 'action': array([3, 3, 3, ..., 3, 3, 3]),\n",
       " 'reward': array([ 0.0457314 ,  0.38802573,  0.4453292 , ..., -0.55317291,\n",
       "        -0.56369207, -0.5698712 ]),\n",
       " 'done': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'terminal': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'info': {},\n",
       " 'pscore': array([1.        , 1.        , 0.99999619, ..., 1.        , 1.        ,\n",
       "        1.        ]),\n",
       " 'behavior_policy': 'ddqn_softmax_tau_1.0',\n",
       " 'dataset_id': 0}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logged_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For offline RL and OPE procedures, please refer to [examples/quickstart/rec/rec_synthetic_discrete_basic.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_discrete_basic.ipynb).\n",
    "\n",
    "For more advanced topic in OPE and OPS, please refer to [examples/quickstart/rec/rec_synthetic_discrete_advanced.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_discrete_advanced.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Collecting Logged Datasets with Multiple Behavior Policies and Random Seeds\n",
    "Finally, we show how to collect logged data by several behavior policies and random seeds.\n",
    "\n",
    "<!-- We show the example in the discrete action case, but the continuous action case can also be handled in a simmilar manner. -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# standardized environment for discrete action\n",
    "env = gym.make(\"RECEnv-v0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define several behavior policies\n",
    "behavior_policy_01 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.1,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.1\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policy_03 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.3,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.3\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policy_05 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.5,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.5\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policy_07 = EpsilonGreedyHead(\n",
    "    ddqn, \n",
    "    n_actions=env.action_space.n,\n",
    "    epsilon=0.7,  # probability of taking random action\n",
    "    name=\"ddqn_eps_0.7\",\n",
    "    random_state=random_state,\n",
    ")\n",
    "behavior_policies = [behavior_policy_01, behavior_policy_03, behavior_policy_05, behavior_policy_07]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize the dataset class\n",
    "dataset = SyntheticDataset(\n",
    "    env=env,\n",
    "    max_episode_steps=env.step_per_episode,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logged_datasets = dataset.obtain_episodes(\n",
    "    behavior_policies=behavior_policies,\n",
    "    n_datasets=2,  # number of random states\n",
    "    n_trajectories=10000, \n",
    "    random_state=random_state,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'size': 100000,\n",
       " 'n_trajectories': 10000,\n",
       " 'step_per_trajectory': 10,\n",
       " 'action_type': 'discrete',\n",
       " 'n_actions': 5,\n",
       " 'action_dim': None,\n",
       " 'action_meaning': None,\n",
       " 'action_keys': None,\n",
       " 'state_dim': 5,\n",
       " 'state_keys': None,\n",
       " 'state': array([[ 0.04479014,  0.11492637, -0.05131316, -0.0280654 ,  0.0347718 ],\n",
       "        [ 0.25961711,  0.85642393, -0.29858954, -0.12037568,  0.30901616],\n",
       "        [ 0.19436985,  0.88078279, -0.22501215, -0.03685871,  0.36667834],\n",
       "        ...,\n",
       "        [ 0.09529246, -0.70234252,  0.10841566, -0.41846472, -0.55746538],\n",
       "        [ 0.10943013, -0.71880701,  0.04250762, -0.40402341, -0.5534435 ],\n",
       "        [ 0.11996917, -0.72904206, -0.00852912, -0.39147998, -0.54843026]]),\n",
       " 'action': array([3, 3, 3, ..., 3, 3, 3]),\n",
       " 'reward': array([ 0.0457314 ,  0.38802573,  0.4453292 , ..., -0.55317291,\n",
       "        -0.56369207, -0.5698712 ]),\n",
       " 'done': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'terminal': array([0., 0., 0., ..., 0., 0., 1.]),\n",
       " 'info': {},\n",
       " 'pscore': array([0.92, 0.92, 0.92, ..., 0.92, 0.92, 0.92]),\n",
       " 'behavior_policy': 'ddqn_eps_0.1',\n",
       " 'dataset_id': 0}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logged_dataset = logged_datasets.get(behavior_policy_name=behavior_policies[0].name, dataset_id=0)\n",
    "logged_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For offline RL and OPE procedures with multiple logged datasets, please refer to [examples/quickstart/rec/rec_synthetic_discrete_multiple.ipynb](https://github.com/hakuhodo-technologies/scope-rl/blob/main/examples/quickstart/rec/rec_synthetic_discrete_multiple.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "- Sarah Dean and Jamie Morgenstern. \\\n",
    "\"Preference Dynamics Under Personalized Recommendations.\", 2022.\n",
    "\n",
    "- Yuta Saito, Shunsuke Aihara, Megumi Matsutani, and Yusuke Narita. \\\n",
    "\"Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation.\", 2021.\n",
    "\n",
    "- Takuma Seno and Michita Imai. \\\n",
    "\"d3rlpy: An Offline Deep Reinforcement Library.\", 2021.\n",
    "\n",
    "- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. \\\n",
    "\"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\" 2018.\n",
    "\n",
    "- Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. \\\n",
    "\"OpenAI Gym.\", 2016.\n",
    "\n",
    "- Hado van Hasselt, Arthur Guez, and David Silver. \\\n",
    "\"Deep Reinforcement Learning with Double Q-learning.\", 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "70404ee114725fce8ed9e697d67827f8546c678889944e6d695790702cbfe1f5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}