lunar_lander_player.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yXsnCWPtWSNk"
   },
   "source": [
    "## Preliminary work\n",
    "\n",
    "First, we need to install all necessary packages.\n",
    "One of them, gym, builded by OpenAI, is a toolkit for developing Reinforcement Learning algorithm. Other packages are for visualization in colab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyopengl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5e2bScpnkVbv",
    "outputId": "f79d9e33-6b97-4abf-f4f5-fd3b3ac7b7ea"
   },
   "outputs": [],
   "source": [
    "!apt update\n",
    "!apt install xvfb -y\n",
    "!pip install gymnasium[box2d] pyvirtualdisplay tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M_-i3cdoYsks"
   },
   "source": [
    "\n",
    "Next, set up virtual display，and import all necessaary packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nl2nREINDLiw"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "from pyvirtualdisplay import Display\n",
    "virtual_display = Display(visible=0, size=(1400, 900))\n",
    "virtual_display.start()\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from IPython import display\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torch.nn.functional as F\n",
    "from torch.distributions import Categorical\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fV9i8i2YkRbO"
   },
   "outputs": [],
   "source": [
    "# Fix the seed to make the behavior deterministic.\n",
    "seed = 543\n",
    "def fix(env, seed):\n",
    "  # according to https://gymnasium.farama.org/api/env/\n",
    "  env.reset(seed=seed)\n",
    "  env.action_space.seed(seed)\n",
    "  torch.manual_seed(seed)\n",
    "  torch.cuda.manual_seed(seed)\n",
    "  torch.cuda.manual_seed_all(seed)\n",
    "  np.random.seed(seed)\n",
    "  random.seed(seed)\n",
    "  torch.use_deterministic_algorithms(True)\n",
    "  torch.backends.cudnn.benchmark = False\n",
    "  torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "He0XDx6bzjgC"
   },
   "source": [
    "Last, call gym and build an [Lunar Lander](https://gym.openai.com/envs/LunarLander-v2/) environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "N_4-xJcbBt09"
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "import gymnasium as gym\n",
    "import random\n",
    "env = gym.make('LunarLander-v2',render_mode='rgb_array')\n",
    "# fix the environment.\n",
    "fix(env, seed) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NrkVvTrvWZ5H"
   },
   "source": [
    "## What Lunar Lander？\n",
    "\n",
    "“LunarLander-v2”is to simulate the situation when the craft lands on the surface of the moon.\n",
    "\n",
    "This task is to enable the craft to land \"safely\" at the pad between the two yellow flags.\n",
    "> Landing pad is always at coordinates (0,0).\n",
    "> Coordinates are the first two numbers in state vector.\n",
    "\n",
    "![](https://gym.openai.com/assets/docs/aeloop-138c89d44114492fd02822303e6b4b07213010bb14ca5856d2d49d6b62d88e53.svg)\n",
    "\n",
    "\"LunarLander-v2\" actually includes \"Agent\" and \"Environment\". \n",
    "\n",
    "In this homework, we will utilize the function `step()` to control the action of \"Agent\". \n",
    "\n",
    "Then `step()` will return the observation/state and reward given by the \"Environment\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mhqp6D-XgHpe"
   },
   "source": [
    "### Random Agent\n",
    "In the end, before we start training, we can see whether a random agent can successfully land the moon or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 269
    },
    "id": "Y3G0bxoccelv",
    "outputId": "07243787-e01a-474d-d014-aca72f0afacb"
   },
   "outputs": [],
   "source": [
    "env.reset()\n",
    "img = plt.imshow(env.render())\n",
    "\n",
    "done = False\n",
    "while not done:\n",
    "    action = env.action_space.sample()\n",
    "    observation, reward, done, _ , _ = env.step(action)\n",
    "\n",
    "    img.set_data(env.render())\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "    print(f'reward:{reward}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F5paWqo7tWL2"
   },
   "source": [
    "## Policy Gradient\n",
    "Now, we can build a simple policy network. The network will return one of action in the action space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "J8tdmeD-tZew"
   },
   "outputs": [],
   "source": [
    "class PolicyGradientNetwork(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(8, 16)\n",
    "        self.fc2 = nn.Linear(16, 16)\n",
    "        self.fc3 = nn.Linear(16, 4)\n",
    "\n",
    "    def forward(self, state):\n",
    "        hid = torch.tanh(self.fc1(state))\n",
    "        hid = torch.tanh(self.fc2(hid))\n",
    "        return F.softmax(self.fc3(hid), dim=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ynbqJrhIFTC3"
   },
   "source": [
    "Then, we need to build a simple agent. The agent will acts according to the output of the policy network above. There are a few things can be done by agent:\n",
    "- `learn()`：update the policy network from log probabilities and rewards.\n",
    "- `sample()`：After receiving observation from the environment, utilize policy network to tell which action to take. The return values of this function includes action and log probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zZo-IxJx286z"
   },
   "outputs": [],
   "source": [
    "from torch.optim.lr_scheduler import StepLR\n",
    "class PolicyGradientAgent():\n",
    "    \n",
    "    def __init__(self, network):\n",
    "        self.network = network\n",
    "        self.optimizer = optim.SGD(self.network.parameters(), lr=0.001)\n",
    "        \n",
    "    def forward(self, state):\n",
    "        return self.network(state)\n",
    "    def learn(self, log_probs, rewards):\n",
    "        loss = (-log_probs * rewards).sum() # You don't need to revise this to pass simple baseline (but you can)\n",
    "\n",
    "        self.optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        self.optimizer.step()\n",
    "        \n",
    "    def sample(self, state):\n",
    "        float_tensor = torch.FloatTensor(state)\n",
    "        action_prob = self.network(float_tensor)\n",
    "        action_dist = Categorical(action_prob)\n",
    "        action = action_dist.sample()\n",
    "        log_prob = action_dist.log_prob(action)\n",
    "        return action.item(), log_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ehPlnTKyRZf9"
   },
   "source": [
    "Lastly, build a network and agent to start training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GfJIvML-RYjL"
   },
   "outputs": [],
   "source": [
    "network = PolicyGradientNetwork()\n",
    "agent = PolicyGradientAgent(network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ouv23glgf5Qt"
   },
   "source": [
    "## Training Agent\n",
    "\n",
    "Now let's start to train our agent.\n",
    "Through taking all the interactions between agent and environment as training data, the policy network can learn from all these attempts,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_reward(rewards, reward):\n",
    "    ripple_factor = reward\n",
    "    for i, value in enumerate(rewards):\n",
    "        ripple_factor = 0.99*ripple_factor\n",
    "        rewards[len(rewards)-1-i] += ripple_factor\n",
    "    rewards.append(reward)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "b45485042bce49c1a70f5cd8fa20d8ed",
      "2faaf34e32fd4b48b8afa6d5dd9874ec",
      "27d6cb7c7702445d9606f32a727cdc00",
      "33101e21164e4905b5d606e41f199f91",
      "17c50fd3ecd84f989b93039e4f89e8be",
      "57969de708bf497cbc0e848c8b697d51",
      "22850a24e3b74161b7b73e86c36ef177",
      "bd990e9e989f4d06aa6eced39499401d",
      "eaa2b0278631474c9b2355a0d2354639",
      "58ebb12b05fa49a69e8d5e003de18eb1",
      "9675476ac2c14735bb3f06916b55bd03"
     ]
    },
    "id": "vg5rxBBaf38_",
    "outputId": "30b76fe8-9a61-48be-941f-2ab7ba642f41"
   },
   "outputs": [],
   "source": [
    "agent.network.train()  # Switch network into training mode \n",
    "EPISODE_PER_BATCH = 5  # update the  agent every 5 episode\n",
    "NUM_BATCH = 500        # totally update the agent for 400 time\n",
    "\n",
    "avg_total_rewards, avg_final_rewards = [], []\n",
    "\n",
    "prg_bar = tqdm(range(NUM_BATCH))\n",
    "for batch in prg_bar:\n",
    "\n",
    "    log_probs, rewards = [], []\n",
    "    total_rewards, final_rewards = [], []\n",
    "\n",
    "    # collect trajectory\n",
    "    for episode in range(EPISODE_PER_BATCH):\n",
    "        print(f'start env reset')\n",
    "        state,_ = env.reset()\n",
    "        print(f'finish env reset')\n",
    "        total_reward, total_step = 0, 0\n",
    "        seq_rewards = []\n",
    "        while True:\n",
    "            action, log_prob = agent.sample(state) # at, log(at|st)\n",
    "            print(f'start step')\n",
    "            next_state, reward, done, _ , _ = env.step(action)\n",
    "            print(f'finish step')\n",
    "            log_probs.append(log_prob) # [log(a1|s1), log(a2|s2), ...., log(at|st)]\n",
    "            state = next_state\n",
    "            total_reward += reward\n",
    "            total_step += 1\n",
    "            # this code is used for passing the medium baseline.\n",
    "            add_reward(seq_rewards, reward)\n",
    "            # boss : implement Actor-Critic\n",
    "            if done:\n",
    "                final_rewards.append(reward)\n",
    "                total_rewards.append(total_reward)\n",
    "                \n",
    "                break\n",
    "        rewards += seq_rewards\n",
    "    print(f'log probs len:{len(log_probs)},rewards len:{len(rewards)}')\n",
    "    print(f\"rewards looks like \", np.shape(rewards))  \n",
    "    #print(f\"log_probs looks like \", np.shape(log_probs))     \n",
    "    # record training process\n",
    "    avg_total_reward = sum(total_rewards) / len(total_rewards)\n",
    "    avg_final_reward = sum(final_rewards) / len(final_rewards)\n",
    "    avg_total_rewards.append(avg_total_reward)\n",
    "    avg_final_rewards.append(avg_final_reward)\n",
    "    prg_bar.set_description(f\"Total: {avg_total_reward: 4.1f}, Final: {avg_final_reward: 4.1f}\")\n",
    "\n",
    "    # update agent\n",
    "    # rewards = np.concatenate(rewards, axis=0)\n",
    "    rewards = (rewards - np.mean(rewards)) / (np.std(rewards) + 1e-9)  # normalize the reward \n",
    "    agent.learn(torch.stack(log_probs), torch.from_numpy(rewards))\n",
    "    print(\"logs prob looks like \", torch.stack(log_probs).size())\n",
    "    print(\"torch.from_numpy(rewards) looks like \", torch.from_numpy(rewards).size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vNb_tuFYhKVK"
   },
   "source": [
    "### Training Result\n",
    "During the training process, we recorded `avg_total_reward`, which represents the average total reward of episodes before updating the policy network.\n",
    "\n",
    "Theoretically, if the agent becomes better, the `avg_total_reward` will increase.\n",
    "The visualization of the training process is shown below:  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "wZYOI8H10SHN",
    "outputId": "3e250312-c647-44c3-f2ef-61f9169786b7"
   },
   "outputs": [],
   "source": [
    "plt.plot(avg_total_rewards)\n",
    "plt.title(\"Total Rewards\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mV5jj4dThz0Y"
   },
   "source": [
    "In addition, `avg_final_reward` represents average final rewards of episodes. To be specific, final rewards is the last reward received in one episode, indicating whether the craft lands successfully or not.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 281
    },
    "id": "txDZ5vlGWz5w",
    "outputId": "d54becbf-8465-4931-e7f4-5f7e6382d17d"
   },
   "outputs": [],
   "source": [
    "plt.plot(avg_final_rewards)\n",
    "plt.title(\"Final Rewards\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u2HaGRVEYGQS"
   },
   "source": [
    "## Testing\n",
    "The testing result will be the average reward of 5 testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 286
    },
    "id": "5yFuUKKRYH73",
    "outputId": "193b2757-a4a4-415c-b57c-09c1e25d9291"
   },
   "outputs": [],
   "source": [
    "fix(env, seed)\n",
    "agent.network.eval()  # set the network into evaluation mode\n",
    "NUM_OF_TEST = 5 # Do not revise this !!!\n",
    "test_total_reward = []\n",
    "action_list = []\n",
    "for i in range(NUM_OF_TEST):\n",
    "  actions = []\n",
    "  state, _ = env.reset()\n",
    "\n",
    "  img = plt.imshow(env.render())\n",
    "\n",
    "  total_reward = 0\n",
    "\n",
    "  done = False\n",
    "  while not done:\n",
    "      action, _ = agent.sample(state)\n",
    "      actions.append(action)\n",
    "      state, reward, done,_, _ = env.step(action)\n",
    "\n",
    "      total_reward += reward\n",
    "\n",
    "      img.set_data(env.render())\n",
    "      display.display(plt.gcf())\n",
    "      display.clear_output(wait=True)\n",
    "      \n",
    "  print(total_reward)\n",
    "  test_total_reward.append(total_reward)\n",
    "\n",
    "  action_list.append(actions) # save the result of testing \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Aex7mcKr0J01",
    "outputId": "219774e5-c80e-4535-8d5a-a566a9a15c73"
   },
   "outputs": [],
   "source": [
    "print(np.mean(test_total_reward))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "leyebGYRpqsF"
   },
   "source": [
    "Action list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_list = [[1,2,3],[1,3]]\n",
    "#np.shape(test_list)\n",
    "print(f'test list shape:{np.shape(test_list)}')\n",
    "print(f'type:{type(test_list)}')\n",
    "print(f'action list type:{type(action_list)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "hGAH4YWDpp4u",
    "outputId": "2d249665-ff47-4b08-9d17-a1d78bdb18f4"
   },
   "outputs": [],
   "source": [
    "print(\"Action list looks like \", action_list)\n",
    "# ecah episode's action list varies, so this call might fail.\n",
    "#print(\"Action list's shape looks like \", np.shape(action_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fNkmwucrHMen"
   },
   "source": [
    "Analysis of actions taken by agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WHdAItjj1nxw",
    "outputId": "f6eac1be-ed9b-4748-e436-e0a154a45268"
   },
   "outputs": [],
   "source": [
    "distribution = {}\n",
    "for actions in action_list:\n",
    "  for action in actions:\n",
    "    if action not in distribution.keys():\n",
    "      distribution[action] = 1\n",
    "    else:\n",
    "      distribution[action] += 1\n",
    "print(distribution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ricE0schY75M"
   },
   "source": [
    "Saving the result of Model Testing\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "GZsMkGmIY42b",
    "outputId": "1867c023-52fc-43a0-f4aa-cb885db52632"
   },
   "outputs": [],
   "source": [
    "PATH = \"Action_List.npy\" # Can be modified into the name or path you want\n",
    "np.save(PATH ,np.array(action_list)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "asK7WfbkaLjt"
   },
   "source": [
    "### This is the file you need to submit !!!\n",
    "Download the testing result to your device\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 17
    },
    "id": "c-CqyhHzaWAL",
    "outputId": "5d5e9170-a642-4103-cad9-bfe89ed82105"
   },
   "outputs": [],
   "source": [
    "from google.colab import files\n",
    "files.download(PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wUBtYXG2eaqf"
   },
   "source": [
    "## Reference\n",
    "\n",
    "Below are some useful tips for you to get high score.\n",
    "\n",
    "- [DRL Lecture 1: Policy Gradient (Review)](https://youtu.be/z95ZYgPgXOY)\n",
    "- [ML Lecture 23-3: Reinforcement Learning (including Q-learning) start at 30:00](https://youtu.be/2-JNBzCq77c?t=1800)\n",
    "- [Lecture 7: Policy Gradient, David Silver](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "17c50fd3ecd84f989b93039e4f89e8be": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "22850a24e3b74161b7b73e86c36ef177": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "27d6cb7c7702445d9606f32a727cdc00": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bd990e9e989f4d06aa6eced39499401d",
      "max": 500,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_eaa2b0278631474c9b2355a0d2354639",
      "value": 500
     }
    },
    "2faaf34e32fd4b48b8afa6d5dd9874ec": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_57969de708bf497cbc0e848c8b697d51",
      "placeholder": "​",
      "style": "IPY_MODEL_22850a24e3b74161b7b73e86c36ef177",
      "value": "Total: -154.0, Final: -100.0: 100%"
     }
    },
    "33101e21164e4905b5d606e41f199f91": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_58ebb12b05fa49a69e8d5e003de18eb1",
      "placeholder": "​",
      "style": "IPY_MODEL_9675476ac2c14735bb3f06916b55bd03",
      "value": " 500/500 [15:18&lt;00:00,  1.64s/it]"
     }
    },
    "57969de708bf497cbc0e848c8b697d51": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "58ebb12b05fa49a69e8d5e003de18eb1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9675476ac2c14735bb3f06916b55bd03": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b45485042bce49c1a70f5cd8fa20d8ed": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2faaf34e32fd4b48b8afa6d5dd9874ec",
       "IPY_MODEL_27d6cb7c7702445d9606f32a727cdc00",
       "IPY_MODEL_33101e21164e4905b5d606e41f199f91"
      ],
      "layout": "IPY_MODEL_17c50fd3ecd84f989b93039e4f89e8be"
     }
    },
    "bd990e9e989f4d06aa6eced39499401d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "eaa2b0278631474c9b2355a0d2354639": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}