Course_OpenAIBaselines_Unit4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using OpenAI with ROS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/robotignite_logo_text.png\" width=\"700\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unit 4: Define MyMovingCubeOneDiskWalkEnv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">SUMMARY</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time to completion: <b>2 hours</b><br><br>\n",
    "In this unit, you will learn how to create a Task Environment for a Moving Cube with a single disk in the roll axis using the **OpenAI ROS** structure. Also, you will use the Qlearn algorithm for training the RoboCube."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">END OF SUMMARY</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have to define the class for our learning episodes. This environment depends directly on what you want the robot to learn, and how."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this environment, we are going to set up everything to induce the robot to learn how to walk forwards through reinforcement learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create your Own Version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, let's add the new script to our package:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"float:left;background: #407EAF\">\n",
    "<tr>\n",
    "<th>\n",
    "<p class=\"transparent\">Execute in WebShell #1</p>\n",
    "</th>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "roscd my_moving_cube_pkg/scripts/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "touch my_one_disk_walk.py;chmod +x my_one_disk_walk.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here you have the template:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gym import spaces\n",
    "import my_robot_env\n",
    "from gym.envs.registration import register\n",
    "import rospy\n",
    "\n",
    "# The path is __init__.py of openai_ros, where we import the MovingCubeOneDiskWalkEnv directly\n",
    "timestep_limit_per_episode = 1000 # Can be any Value\n",
    "\n",
    "register(\n",
    "        id='MyTrainingEnv-v0',\n",
    "        entry_point='template_my_training_env:MovingCubeOneDiskWalkEnv',\n",
    "        timestep_limit=timestep_limit_per_episode,\n",
    "    )\n",
    "\n",
    "class MyTrainingEnv(cube_single_disk_env.MyRobotEnv):\n",
    "    def __init__(self):\n",
    "        \n",
    "        # Only variable needed to be set here\n",
    "        number_actions = rospy.get_param('/my_robot_namespace/n_actions')\n",
    "        self.action_space = spaces.Discrete(number_actions)\n",
    "        \n",
    "        # This is the most common case of Box observation type\n",
    "        high = numpy.array([\n",
    "            obs1_max_value,\n",
    "            obs12_max_value,\n",
    "            ...\n",
    "            obsN_max_value\n",
    "            ])\n",
    "            \n",
    "        self.observation_space = spaces.Box(-high, high)\n",
    "        \n",
    "        # Variables that we retrieve through the param server, loded when launch training launch.\n",
    "        \n",
    "\n",
    "\n",
    "        # Here we will add any init functions prior to starting the MyRobotEnv\n",
    "        super(MyTrainingEnv, self).__init__()\n",
    "\n",
    "\n",
    "    def _set_init_pose(self):\n",
    "        \"\"\"Sets the Robot in its init pose\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "\n",
    "    def _init_env_variables(self):\n",
    "        \"\"\"\n",
    "        Inits variables needed to be initialised each time we reset at the start\n",
    "        of an episode.\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "\n",
    "\n",
    "    def _set_action(self, action):\n",
    "        \"\"\"\n",
    "        Move the robot based on the action variable given\n",
    "        \"\"\"\n",
    "        # TODO: Move robot\n",
    "\n",
    "    def _get_obs(self):\n",
    "        \"\"\"\n",
    "        Here we define what sensor data of our robots observations\n",
    "        To know which Variables we have acces to, we need to read the\n",
    "        MyRobotEnv API DOCS\n",
    "        :return: observations\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "        return observations\n",
    "\n",
    "    def _is_done(self, observations):\n",
    "        \"\"\"\n",
    "        Decide if episode is done based on the observations\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "        return done\n",
    "\n",
    "    def _compute_reward(self, observations, done):\n",
    "        \"\"\"\n",
    "        Return the reward based on the observations given\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "        return reward\n",
    "        \n",
    "    # Internal TaskEnv Methods\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Import the RobotEnv you want it to inherit from"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we want it to inherit from the RobotEnv we created in the previous unit, so we have to import the **MyCubeSingleDiskEnv** from the python module **my_cube_single_disk_env.py**. So, we convert:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import my_robot_env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Into:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import my_cube_single_disk_env"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Register the new TaskEnv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to be able to use a gym environment, we need to register it. This is done like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "register(\n",
    "        id='MyTrainingEnv-v0',\n",
    "        entry_point='my_robot_env:MovingCubeOneDiskWalkEnv',\n",
    "        timestep_limit=timestep_limit_per_episode,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we have to change it to import from the **correct python module**, give it an **Id**, and set the **timesteps_limit** per episode."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the python module where we have to import it from is exactly the one we are defining it as, so **my_one_disk_walk**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "timestep_limit_per_episode = 1000\n",
    "\n",
    "register(\n",
    "        id='MyMovingCubeOneDiskWalkEnv-v0',\n",
    "        entry_point='my_one_disk_walk:MyMovingCubeOneDiskWalkEnv',\n",
    "        timestep_limit=timestep_limit_per_episode,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's custom to put the same name for the **ID** and the **TaskEnv**, but you can put anything you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Initialize the TaskEnv Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class MyMovingCubeOneDiskWalkEnv(my_cube_single_disk_env.MyCubeSingleDiskEnv):\n",
    "    def __init__(self):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we are telling it to inherit from the class **MyCubeSingleDiskEnv**, from the python module that we created in a previous unit called **my_cube_single_disk_env.py**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we set the number of actions that will have this learning. This is the only thing that you have to set related to **Gym Environments**. It has to be set here because it's directly dependent on what and how you want the robot to learn something."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Only variable needed to be set here\n",
    "number_actions = rospy.get_param('/moving_cube/n_actions')\n",
    "self.action_space = spaces.Discrete(number_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we are retrieving it from the param server of ROS, but if you don't want to have to upload a yaml file to the param server, you could simply do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Only variable needed to be set here\n",
    "number_actions = 4 # Any value\n",
    "self.action_space = spaces.Discrete(number_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we highly recommend using a yaml with all your configuration data, as it will make life much easier and allow you to test different configurations in remote systems with ease."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have to create the **observations_space**. In this case, we need it to be a **Box** type because we need a range of floats that are different for each of the observations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This is the most common case of Box observation type\n",
    "high = numpy.array([\n",
    "    obs1_max_value,\n",
    "    obs12_max_value,\n",
    "    ...\n",
    "    obsN_max_value\n",
    "    ])\n",
    "\n",
    "self.observation_space = spaces.Box(-high, high)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we extract it from the param server loaded variables for the most part:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Actions and Observations\n",
    "self.roll_speed_fixed_value = rospy.get_param('/moving_cube/roll_speed_fixed_value')\n",
    "self.roll_speed_increment_value = rospy.get_param('/moving_cube/roll_speed_increment_value')\n",
    "self.max_distance = rospy.get_param('/moving_cube/max_distance')\n",
    "max_roll = 2 * math.pi\n",
    "self.max_pitch_angle = rospy.get_param('/moving_cube/max_pitch_angle')\n",
    "self.max_y_linear_speed = rospy.get_param('/moving_cube/max_y_linear_speed')\n",
    "self.max_yaw_angle = rospy.get_param('/moving_cube/max_yaw_angle')\n",
    "\n",
    "high = numpy.array([\n",
    "            self.roll_speed_fixed_value,\n",
    "            self.max_distance,\n",
    "            max_roll,\n",
    "            self.max_pitch_angle,\n",
    "            self.max_y_linear_speed,\n",
    "            self.max_y_linear_speed,\n",
    "            ])\n",
    "        \n",
    "self.observation_space = spaces.Box(-high, high)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will return an array of **six** different values in the **_get_obs** function that define the robot's state. More on that later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we call the **__init__** method of the parent class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "super(MyMovingCubeOneDiskWalkEnv, self).__init__()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Fill in the Learning Functions used by the GrandParent Class RobotGazeboEnv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to fill these functions that are used by the **RobotGazeboEnv** to execute different aspects of Gym and are needed for the Learning Script to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "   def _set_init_pose(self):\n",
    "        \"\"\"Sets the Robot in its init pose\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "    \n",
    "    def _init_env_variables(self):\n",
    "        \"\"\"\n",
    "        Inits variables needed to be initialised each time we reset at the start\n",
    "        of an episode.\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "\n",
    "\n",
    "    def _set_action(self, action):\n",
    "        \"\"\"\n",
    "        Move the robot based on the action variable given\n",
    "        \"\"\"\n",
    "        # TODO: Move robot\n",
    "\n",
    "    def _get_obs(self):\n",
    "        \"\"\"\n",
    "        Here we define what sensor data of our robot's observations\n",
    "        To know which Variables we have access to, we need to read the\n",
    "        MyRobotEnv API DOCS\n",
    "        :return: observations\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "        return observations\n",
    "\n",
    "    def _is_done(self, observations):\n",
    "        \"\"\"\n",
    "        Decide if episode is done based on the observations\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "        return done\n",
    "\n",
    "    def _compute_reward(self, observations, done):\n",
    "        \"\"\"\n",
    "        Return the reward based on the observations given\n",
    "        \"\"\"\n",
    "        # TODO\n",
    "        return reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.0 Set Init Pos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first declare the function that states the init pose in which the robot will start; in this case, it's the one that will decide what speed it will have at the start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    def _set_init_pose(self):\n",
    "        \"\"\"Sets the Robot in its init pose\n",
    "        \"\"\"\n",
    "        self.move_joints(self.init_roll_vel)\n",
    "\n",
    "        return True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And add the **init_roll_vel** value import from the ROS param server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "self.init_roll_vel = rospy.get_param(\"/moving_cube/init_roll_vel\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1 InitEnvVariables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method is called each time we reset an episode. Here we reset the TaskEnv variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _init_env_variables(self):\n",
    "    \"\"\"\n",
    "    Inits variables needed to be initialised each time we reset at the start\n",
    "    of an episode.\n",
    "    :return:\n",
    "    \"\"\"\n",
    "    self.total_distance_moved = 0.0\n",
    "    self.current_y_distance = self.get_y_dir_distance_from_start_point(self.start_point)\n",
    "    self.roll_turn_speed = rospy.get_param('/moving_cube/init_roll_vel')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you can see something interesting. You are using a function that will NOT be defined here in the **TrainingEnv/MyMovingCubeOneDiskWalkEnv**. This method is defined in the **RobotEnv/MyCubeSingleDiskEnv**. This is because it has to do with retrieving sensor data and ROS related stuff, and has nothing to do with the learning of the Task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 Set Action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here you decide how the different actions will be transformed into real Movements, and the command to execute those movements sent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _set_action(self, action):\n",
    "\n",
    "    # We convert the actions to speed movements to send to the parent class CubeSingleDiskEnv\n",
    "    if action == 0:# Move Speed Wheel Forwards\n",
    "        self.roll_turn_speed = self.roll_speed_fixed_value\n",
    "    elif action == 1:# Move Speed Wheel Backwards\n",
    "        self.roll_turn_speed = -self.roll_speed_fixed_value\n",
    "    elif action == 2:# Stop Speed Wheel\n",
    "        self.roll_turn_speed = 0.0\n",
    "    elif action == 3:# Increment Speed\n",
    "        self.roll_turn_speed += self.roll_speed_increment_value\n",
    "    elif action == 4:# Decrement Speed\n",
    "        self.roll_turn_speed -= self.roll_speed_increment_value\n",
    "\n",
    "    # We clamp Values to maximum\n",
    "    rospy.logdebug(\"roll_turn_speed before clamp==\"+str(self.roll_turn_speed))\n",
    "    self.roll_turn_speed = numpy.clip(self.roll_turn_speed,\n",
    "                                      -self.roll_speed_fixed_value,\n",
    "                                      self.roll_speed_fixed_value)\n",
    "    rospy.logdebug(\"roll_turn_speed after clamp==\" + str(self.roll_turn_speed))\n",
    "\n",
    "    # We tell the OneDiskCube to spin the RollDisk at the selected speed\n",
    "    self.move_joints(self.roll_turn_speed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and we add to the __init__ method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Variables that we retrieve through the param server, loaded when launch training launch.\n",
    "self.roll_speed_fixed_value = rospy.get_param('/moving_cube/roll_speed_fixed_value')\n",
    "self.roll_speed_increment_value = rospy.get_param('/moving_cube/roll_speed_increment_value')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we consider that the Cube can perform **FIVE** actions, which are: Move Speed Wheel Forwards, Move Speed Wheel Backwards, Stop Speed Wheel, Increment Speed, and Decrement Speed. This way it will be able to ramp the speed up/down slowly and then increment it in bursts. This is vital for creating the momentum differences that make the cube move. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.3 Get Observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we retrieve sensor data using our parent class **CubeSingleDiskEnv** to get all the sensor data, and then we process it to return the observations that we see fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _get_obs(self):\n",
    "        \"\"\"\n",
    "        Here we define what sensor data defines our robots observations\n",
    "        To know which Variables we have access to, we need to read the\n",
    "        MyCubeSingleDiskEnv API DOCS\n",
    "        :return:\n",
    "        \"\"\"\n",
    "\n",
    "        # We get the orientation of the cube in RPY\n",
    "        roll, pitch, yaw = self.get_orientation_euler()\n",
    "\n",
    "        # We get the distance from the origin\n",
    "        #distance = self.get_distance_from_start_point(self.start_point)\n",
    "        y_distance = self.get_y_dir_distance_from_start_point(self.start_point)\n",
    "\n",
    "        # We get the current speed of the Roll Disk\n",
    "        current_disk_roll_vel = self.get_roll_velocity()\n",
    "\n",
    "        # We get the linear speed in the y axis\n",
    "        y_linear_speed = self.get_y_linear_speed()\n",
    "\n",
    "        cube_observations = [\n",
    "            round(current_disk_roll_vel, 0),\n",
    "            #round(distance, 1),\n",
    "            round(y_distance, 1),\n",
    "            round(roll, 1),\n",
    "            round(pitch, 1),\n",
    "            round(y_linear_speed,1),\n",
    "            round(yaw, 1),\n",
    "        ]\n",
    "\n",
    "        return cube_observations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use internal functions that we will define later, which return the robot's sensory data, already processed for us. In this case, we leave only one decimal in the sensor data because it's enough for our purposes, making the defining state for the Qlearn algorithm or whatever faster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also have to add some imports of parameters in the **init__** function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "self.start_point = Point()\n",
    "self.start_point.x = rospy.get_param(\"/moving_cube/init_cube_pose/x\")\n",
    "self.start_point.y = rospy.get_param(\"/moving_cube/init_cube_pose/y\")\n",
    "self.start_point.z = rospy.get_param(\"/moving_cube/init_cube_pose/z\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And an import:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from geometry_msgs.msg import Point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.4 Is Done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on observation, we decide if the learning episode is over. In this case, we consider the episode done when the pitch angle surpasses a certain threshold, which we get from the rosparam. We also consider the episode done when the yaw exceeds the maximum angle. This way, we don't waste steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _is_done(self, observations):\n",
    "\n",
    "        pitch_angle = observations[3]\n",
    "        yaw_angle = observations[5]\n",
    "\n",
    "        if abs(pitch_angle) > self.max_pitch_angle:\n",
    "            rospy.logerr(\"WRONG Cube Pitch Orientation==>\" + str(pitch_angle))\n",
    "            done = True\n",
    "        else:\n",
    "            rospy.logdebug(\"Cube Pitch Orientation Ok==>\" + str(pitch_angle))\n",
    "            if abs(yaw_angle) > self.max_yaw_angle:\n",
    "                rospy.logerr(\"WRONG Cube Yaw Orientation==>\" + str(yaw_angle))\n",
    "                done = True\n",
    "            else:\n",
    "                rospy.logdebug(\"Cube Yaw Orientation Ok==>\" + str(yaw_angle))\n",
    "                done = False\n",
    "\n",
    "        return done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we also have to add the import of the parameters in the init:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "self.max_pitch_angle = rospy.get_param('/moving_cube/max_pitch_angle')\n",
    "self.max_yaw_angle = rospy.get_param('/moving_cube/max_yaw_angle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.5 Compute Reward:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is probably one of the most important methods. Here you will condition how the robot will learn by rewarding good-result actions and punishing bad practices. This has an enormous effect on emerging behaviours that the robot will have while trying to solve the task at hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def _compute_reward(self, observations, done):\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            y_distance_now = observations[1]\n",
    "            delta_distance = y_distance_now - self.current_y_distance\n",
    "            rospy.logdebug(\"y_distance_now=\" + str(y_distance_now)+\", current_y_distance=\" + str(self.current_y_distance))\n",
    "            rospy.logdebug(\"delta_distance=\" + str(delta_distance))\n",
    "            reward_distance = delta_distance * self.move_distance_reward_weight\n",
    "            self.current_y_distance = y_distance_now\n",
    "\n",
    "            y_linear_speed = observations[4]\n",
    "            rospy.logdebug(\"y_linear_speed=\" + str(y_linear_speed))\n",
    "            reward_y_axis_speed = y_linear_speed * self.y_linear_speed_reward_weight\n",
    "\n",
    "            # Negative Reward for yaw different from zero.\n",
    "            yaw_angle = observations[5]\n",
    "            rospy.logdebug(\"yaw_angle=\" + str(yaw_angle))\n",
    "            # Worst yaw is 90 and 270 degrees, best 0 and 180. We use sin function for giving reward.\n",
    "            sin_yaw_angle = math.sin(yaw_angle)\n",
    "            rospy.logdebug(\"sin_yaw_angle=\" + str(sin_yaw_angle))\n",
    "            reward_y_axis_angle = -1 * abs(sin_yaw_angle) * self.y_axis_angle_reward_weight\n",
    "\n",
    "\n",
    "            # We are not intereseted in decimals of the reward, doesnt give any advatage.\n",
    "            reward = round(reward_distance, 0) + round(reward_y_axis_speed, 0) + round(reward_y_axis_angle, 0)\n",
    "            rospy.logdebug(\"reward_distance=\" + str(reward_distance))\n",
    "            rospy.logdebug(\"reward_y_axis_speed=\" + str(reward_y_axis_speed))\n",
    "            rospy.logdebug(\"reward_y_axis_angle=\" + str(reward_y_axis_angle))\n",
    "            rospy.logdebug(\"reward=\" + str(reward))\n",
    "        else:\n",
    "            reward = -self.end_episode_points\n",
    "\n",
    "        return reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you will have to import some parameters from ROSPARAM server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "self.move_distance_reward_weight = rospy.get_param(\"/moving_cube/move_distance_reward_weight\")\n",
    "self.y_linear_speed_reward_weight = rospy.get_param(\"/moving_cube/y_linear_speed_reward_weight\")\n",
    "self.y_axis_angle_reward_weight = rospy.get_param(\"/moving_cube/y_axis_angle_reward_weight\")\n",
    "self.end_episode_points = rospy.get_param(\"/moving_cube/end_episode_points\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explain the rewards used, let's have a look at the way the total reward is calculated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reward = round(reward_distance, 0) + round(reward_y_axis_speed, 0) + round(reward_y_axis_angle, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Total rewards is the sum of three subrewards:\n",
    "* reward_distance: Rewards given when there is an increase from a previous step of the distance from the start point. This incourages the cube to keep moving.\n",
    "* reward_y_axis_speed: We give points for moving forwards on the Y-axis. This conditions the robot cube to go forwards.\n",
    "* reward_y_axis_angle: We give negative points based on how much the cube deviates from going in a straight line following the Y-axis. We use the sine function because the worst configuration is a 90/-90 degree devuation; after that, if it's backwards, it just has to learn to move the other way round to move positive on the Y-AXIS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And don't forget to add the imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.6 We Add the internal functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now add the internal functions that we need to access the sensor data of the RobotEnv parent, and process that data to give more rich data to the observation, reward, and done methods here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    # Internal TaskEnv Methods\n",
    "    def get_y_dir_distance_from_start_point(self, start_point):\n",
    "        \"\"\"\n",
    "        Calculates the distance from the given point and the current position\n",
    "        given by odometry. In this case the increase or decrease in y.\n",
    "        :param start_point:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        y_dist_dir = self.odom.pose.pose.position.y - start_point.y\n",
    "    \n",
    "        return y_dist_dir\n",
    "    \n",
    "    \n",
    "    def get_distance_from_start_point(self, start_point):\n",
    "        \"\"\"\n",
    "        Calculates the distance from the given point and the current position\n",
    "        given by odometry\n",
    "        :param start_point:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        distance = self.get_distance_from_point(start_point,\n",
    "                                                self.odom.pose.pose.position)\n",
    "    \n",
    "        return distance\n",
    "    \n",
    "    def get_distance_from_point(self, pstart, p_end):\n",
    "        \"\"\"\n",
    "        Given a Vector3 Object, get distance from current position\n",
    "        :param p_end:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        a = numpy.array((pstart.x, pstart.y, pstart.z))\n",
    "        b = numpy.array((p_end.x, p_end.y, p_end.z))\n",
    "    \n",
    "        distance = numpy.linalg.norm(a - b)\n",
    "    \n",
    "        return distance\n",
    "    \n",
    "    def get_orientation_euler(self):\n",
    "        # We convert from quaternions to euler\n",
    "        orientation_list = [self.odom.pose.pose.orientation.x,\n",
    "                            self.odom.pose.pose.orientation.y,\n",
    "                            self.odom.pose.pose.orientation.z,\n",
    "                            self.odom.pose.pose.orientation.w]\n",
    "    \n",
    "        roll, pitch, yaw = euler_from_quaternion(orientation_list)\n",
    "        return roll, pitch, yaw\n",
    "    \n",
    "    def get_roll_velocity(self):\n",
    "        # We get the current joint roll velocity\n",
    "        roll_vel = self.joints.velocity[0]\n",
    "        return roll_vel\n",
    "    \n",
    "    def get_y_linear_speed(self):\n",
    "        # We get the current joint roll velocity\n",
    "        y_linear_speed = self.odom.twist.twist.linear.y\n",
    "        return y_linear_speed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, not even the **get_odom** method, for example, is needed because its getting the data directly from the odom variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't forget the imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tf.transformations import euler_from_quaternion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final Script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should have something similar to this as the final TaskEnv:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**my_one_disk_walk.py**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import rospy\n",
    "import numpy\n",
    "import math\n",
    "from gym import spaces\n",
    "import my_cube_single_disk_env\n",
    "from gym.envs.registration import register\n",
    "from geometry_msgs.msg import Point\n",
    "from tf.transformations import euler_from_quaternion\n",
    "\n",
    "timestep_limit_per_episode = 10000 # Can be any Value\n",
    "\n",
    "register(\n",
    "        id='MyMovingCubeOneDiskWalkEnv-v0',\n",
    "        entry_point='my_one_disk_walk:MyMovingCubeOneDiskWalkEnv',\n",
    "        timestep_limit=timestep_limit_per_episode,\n",
    "    )\n",
    "\n",
    "class MyMovingCubeOneDiskWalkEnv(my_cube_single_disk_env.MyCubeSingleDiskEnv):\n",
    "    def __init__(self):\n",
    "        \n",
    "        # Only variable needed to be set here\n",
    "        number_actions = rospy.get_param('/moving_cube/n_actions')\n",
    "        self.action_space = spaces.Discrete(number_actions)\n",
    "        \n",
    "        \n",
    "        #number_observations = rospy.get_param('/moving_cube/n_observations')\n",
    "        \"\"\"\n",
    "        We set the Observation space for the 6 observations\n",
    "        cube_observations = [\n",
    "            round(current_disk_roll_vel, 0),\n",
    "            round(y_distance, 1),\n",
    "            round(roll, 1),\n",
    "            round(pitch, 1),\n",
    "            round(y_linear_speed,1),\n",
    "            round(yaw, 1),\n",
    "        ]\n",
    "        \"\"\"\n",
    "        \n",
    "        # Actions and Observations\n",
    "        self.roll_speed_fixed_value = rospy.get_param('/moving_cube/roll_speed_fixed_value')\n",
    "        self.roll_speed_increment_value = rospy.get_param('/moving_cube/roll_speed_increment_value')\n",
    "        self.max_distance = rospy.get_param('/moving_cube/max_distance')\n",
    "        max_roll = 2 * math.pi\n",
    "        self.max_pitch_angle = rospy.get_param('/moving_cube/max_pitch_angle')\n",
    "        self.max_y_linear_speed = rospy.get_param('/moving_cube/max_y_linear_speed')\n",
    "        self.max_yaw_angle = rospy.get_param('/moving_cube/max_yaw_angle')\n",
    "        \n",
    "        \n",
    "        high = numpy.array([\n",
    "            self.roll_speed_fixed_value,\n",
    "            self.max_distance,\n",
    "            max_roll,\n",
    "            self.max_pitch_angle,\n",
    "            self.max_y_linear_speed,\n",
    "            self.max_y_linear_speed,\n",
    "            ])\n",
    "        \n",
    "        self.observation_space = spaces.Box(-high, high)\n",
    "        \n",
    "        rospy.logwarn(\"ACTION SPACES TYPE===>\"+str(self.action_space))\n",
    "        rospy.logwarn(\"OBSERVATION SPACES TYPE===>\"+str(self.observation_space))\n",
    "        \n",
    "        # Variables that we retrieve through the param server, loded when launch training launch.\n",
    "        self.init_roll_vel = rospy.get_param(\"/moving_cube/init_roll_vel\")\n",
    "        \n",
    "        \n",
    "        # Get Observations\n",
    "        self.start_point = Point()\n",
    "        self.start_point.x = rospy.get_param(\"/moving_cube/init_cube_pose/x\")\n",
    "        self.start_point.y = rospy.get_param(\"/moving_cube/init_cube_pose/y\")\n",
    "        self.start_point.z = rospy.get_param(\"/moving_cube/init_cube_pose/z\")\n",
    "        \n",
    "        # Rewards\n",
    "        self.move_distance_reward_weight = rospy.get_param(\"/moving_cube/move_distance_reward_weight\")\n",
    "        self.y_linear_speed_reward_weight = rospy.get_param(\"/moving_cube/y_linear_speed_reward_weight\")\n",
    "        self.y_axis_angle_reward_weight = rospy.get_param(\"/moving_cube/y_axis_angle_reward_weight\")\n",
    "        self.end_episode_points = rospy.get_param(\"/moving_cube/end_episode_points\")\n",
    "\n",
    "        self.cumulated_steps = 0.0\n",
    "\n",
    "        # Here we will add any init functions prior to starting the MyRobotEnv\n",
    "        super(MyMovingCubeOneDiskWalkEnv, self).__init__()\n",
    "\n",
    "    def _set_init_pose(self):\n",
    "        \"\"\"Sets the Robot in its init pose\n",
    "        \"\"\"\n",
    "        self.move_joints(self.init_roll_vel)\n",
    "\n",
    "        return True\n",
    "\n",
    "\n",
    "    def _init_env_variables(self):\n",
    "        \"\"\"\n",
    "        Inits variables needed to be initialised each time we reset at the start\n",
    "        of an episode.\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        self.total_distance_moved = 0.0\n",
    "        self.current_y_distance = self.get_y_dir_distance_from_start_point(self.start_point)\n",
    "        self.roll_turn_speed = rospy.get_param('/moving_cube/init_roll_vel')\n",
    "        # For Info Purposes\n",
    "        self.cumulated_reward = 0.0\n",
    "        #self.cumulated_steps = 0.0\n",
    "\n",
    "\n",
    "    def _set_action(self, action):\n",
    "\n",
    "        # We convert the actions to speed movements to send to the parent class CubeSingleDiskEnv\n",
    "        if action == 0:# Move Speed Wheel Forwards\n",
    "            self.roll_turn_speed = self.roll_speed_fixed_value\n",
    "        elif action == 1:# Move Speed Wheel Backwards\n",
    "            self.roll_turn_speed = -1*self.roll_speed_fixed_value\n",
    "        elif action == 2:# Stop Speed Wheel\n",
    "            self.roll_turn_speed = 0.0\n",
    "        elif action == 3:# Increment Speed\n",
    "            self.roll_turn_speed += self.roll_speed_increment_value\n",
    "        elif action == 4:# Decrement Speed\n",
    "            self.roll_turn_speed -= self.roll_speed_increment_value\n",
    "\n",
    "        # We clamp Values to maximum\n",
    "        rospy.logdebug(\"roll_turn_speed before clamp==\"+str(self.roll_turn_speed))\n",
    "        self.roll_turn_speed = numpy.clip(self.roll_turn_speed,\n",
    "                                          -1*self.roll_speed_fixed_value,\n",
    "                                          self.roll_speed_fixed_value)\n",
    "        rospy.logdebug(\"roll_turn_speed after clamp==\" + str(self.roll_turn_speed))\n",
    "\n",
    "        # We tell the OneDiskCube to spin the RollDisk at the selected speed\n",
    "        self.move_joints(self.roll_turn_speed)\n",
    "\n",
    "    def _get_obs(self):\n",
    "        \"\"\"\n",
    "        Here we define what sensor data defines our robots observations\n",
    "        To know which Variables we have acces to, we need to read the\n",
    "        MyCubeSingleDiskEnv API DOCS\n",
    "        :return:\n",
    "        \"\"\"\n",
    "\n",
    "        # We get the orientation of the cube in RPY\n",
    "        roll, pitch, yaw = self.get_orientation_euler()\n",
    "\n",
    "        # We get the distance from the origin\n",
    "        #distance = self.get_distance_from_start_point(self.start_point)\n",
    "        y_distance = self.get_y_dir_distance_from_start_point(self.start_point)\n",
    "\n",
    "        # We get the current speed of the Roll Disk\n",
    "        current_disk_roll_vel = self.get_roll_velocity()\n",
    "\n",
    "        # We get the linear speed in the y axis\n",
    "        y_linear_speed = self.get_y_linear_speed()\n",
    "\n",
    "        \n",
    "        cube_observations = [\n",
    "            round(current_disk_roll_vel, 0),\n",
    "            round(y_distance, 1),\n",
    "            round(roll, 1),\n",
    "            round(pitch, 1),\n",
    "            round(y_linear_speed,1),\n",
    "            round(yaw, 1)\n",
    "        ]\n",
    "        \n",
    "        rospy.logdebug(\"Observations==>\"+str(cube_observations))\n",
    "\n",
    "        return cube_observations\n",
    "        \n",
    "\n",
    "    def _is_done(self, observations):\n",
    "\n",
    "        pitch_angle = observations[3]\n",
    "        yaw_angle = observations[5]\n",
    "\n",
    "        if abs(pitch_angle) > self.max_pitch_angle:\n",
    "            rospy.logerr(\"WRONG Cube Pitch Orientation==>\" + str(pitch_angle))\n",
    "            done = True\n",
    "        else:\n",
    "            rospy.logdebug(\"Cube Pitch Orientation Ok==>\" + str(pitch_angle))\n",
    "            if abs(yaw_angle) > self.max_yaw_angle:\n",
    "                rospy.logerr(\"WRONG Cube Yaw Orientation==>\" + str(yaw_angle))\n",
    "                done = True\n",
    "            else:\n",
    "                rospy.logdebug(\"Cube Yaw Orientation Ok==>\" + str(yaw_angle))\n",
    "                done = False\n",
    "\n",
    "        return done\n",
    "\n",
    "    def _compute_reward(self, observations, done):\n",
    "\n",
    "        if not done:\n",
    "\n",
    "            y_distance_now = observations[1]\n",
    "            delta_distance = y_distance_now - self.current_y_distance\n",
    "            rospy.logdebug(\"y_distance_now=\" + str(y_distance_now)+\", current_y_distance=\" + str(self.current_y_distance))\n",
    "            rospy.logdebug(\"delta_distance=\" + str(delta_distance))\n",
    "            reward_distance = delta_distance * self.move_distance_reward_weight\n",
    "            self.current_y_distance = y_distance_now\n",
    "\n",
    "            y_linear_speed = observations[4]\n",
    "            rospy.logdebug(\"y_linear_speed=\" + str(y_linear_speed))\n",
    "            reward_y_axis_speed = y_linear_speed * self.y_linear_speed_reward_weight\n",
    "\n",
    "            # Negative Reward for yaw different from zero.\n",
    "            yaw_angle = observations[5]\n",
    "            rospy.logdebug(\"yaw_angle=\" + str(yaw_angle))\n",
    "            # Worst yaw is 90 and 270 degrees, best 0 and 180. We use sin function for giving reward.\n",
    "            sin_yaw_angle = math.sin(yaw_angle)\n",
    "            rospy.logdebug(\"sin_yaw_angle=\" + str(sin_yaw_angle))\n",
    "            reward_y_axis_angle = -1 * abs(sin_yaw_angle) * self.y_axis_angle_reward_weight\n",
    "\n",
    "\n",
    "            # We are not intereseted in decimals of the reward, doesn't give any advatage.\n",
    "            reward = round(reward_distance, 0) + round(reward_y_axis_speed, 0) + round(reward_y_axis_angle, 0)\n",
    "            rospy.logdebug(\"reward_distance=\" + str(reward_distance))\n",
    "            rospy.logdebug(\"reward_y_axis_speed=\" + str(reward_y_axis_speed))\n",
    "            rospy.logdebug(\"reward_y_axis_angle=\" + str(reward_y_axis_angle))\n",
    "            rospy.logdebug(\"reward=\" + str(reward))\n",
    "        else:\n",
    "            reward = -1*self.end_episode_points\n",
    "\n",
    "\n",
    "        self.cumulated_reward += reward\n",
    "        rospy.logdebug(\"Cumulated_reward=\" + str(self.cumulated_reward))\n",
    "        self.cumulated_steps += 1\n",
    "        rospy.logdebug(\"Cumulated_steps=\" + str(self.cumulated_steps))\n",
    "        \n",
    "        return reward\n",
    "\n",
    "\n",
    "    # Internal TaskEnv Methods\n",
    "    def get_y_dir_distance_from_start_point(self, start_point):\n",
    "        \"\"\"\n",
    "        Calculates the distance from the given point and the current position\n",
    "        given by odometry. In this case the increase or decrease in y.\n",
    "        :param start_point:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        y_dist_dir = self.odom.pose.pose.position.y - start_point.y\n",
    "    \n",
    "        return y_dist_dir\n",
    "    \n",
    "    \n",
    "    def get_distance_from_start_point(self, start_point):\n",
    "        \"\"\"\n",
    "        Calculates the distance from the given point and the current position\n",
    "        given by odometry\n",
    "        :param start_point:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        distance = self.get_distance_from_point(start_point,\n",
    "                                                self.odom.pose.pose.position)\n",
    "    \n",
    "        return distance\n",
    "    \n",
    "    def get_distance_from_point(self, pstart, p_end):\n",
    "        \"\"\"\n",
    "        Given a Vector3 Object, get distance from current position\n",
    "        :param p_end:\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        a = numpy.array((pstart.x, pstart.y, pstart.z))\n",
    "        b = numpy.array((p_end.x, p_end.y, p_end.z))\n",
    "    \n",
    "        distance = numpy.linalg.norm(a - b)\n",
    "    \n",
    "        return distance\n",
    "    \n",
    "    def get_orientation_euler(self):\n",
    "        # We convert from quaternions to euler\n",
    "        orientation_list = [self.odom.pose.pose.orientation.x,\n",
    "                            self.odom.pose.pose.orientation.y,\n",
    "                            self.odom.pose.pose.orientation.z,\n",
    "                            self.odom.pose.pose.orientation.w]\n",
    "    \n",
    "        roll, pitch, yaw = euler_from_quaternion(orientation_list)\n",
    "        return roll, pitch, yaw\n",
    "    \n",
    "    def get_roll_velocity(self):\n",
    "        # We get the current joint roll velocity\n",
    "        roll_vel = self.joints.velocity[0]\n",
    "        return roll_vel\n",
    "    \n",
    "    def get_y_linear_speed(self):\n",
    "        # We get the current joint roll velocity\n",
    "        y_linear_speed = self.odom.twist.twist.linear.y\n",
    "        return y_linear_speed\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**END my_one_disk_walk.py**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#FF5E00;color:white;\">**TEST new Environment**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are going to use the same scripts that you used for the **CartPole example** to train through Qlearn, but adapting them to use the Cube environments we have just created. We will only need to change the Environments we use, the configuration yaml file, and some minor elements to adapt to the change of the environments. Let's comment only on the elements that will need to be changed in the **start_training.py** file:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1.1 Where to import the Environment from"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing will only have the effect of triggering the register environment method at the top of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import our training environment\n",
    "from openai_ros.task_envs.cartpole_stay_up import stay_up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change it to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import our training environment\n",
    "import my_one_disk_walk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Use the new Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create the Gym environment\n",
    "env = gym.make('CartPoleStayUp-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change it to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env = gym.make('MyMovingCubeOneDiskWalkEnv-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3 Change the Package where logs and robot_namespace for parameters are written"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the logging system\n",
    "rospack = rospkg.RosPack()\n",
    "pkg_path = rospack.get_path('cartpole_v0_training')\n",
    "outdir = pkg_path + '/training_results'\n",
    "env = wrappers.Monitor(env, outdir, force=True) \n",
    "rospy.loginfo ( \"Monitor Wrapper started\")\n",
    "\n",
    "last_time_steps = numpy.ndarray(0)\n",
    "\n",
    "# Loads parameters from the ROS param server\n",
    "# Parameters are stored in a yaml file inside the config directory\n",
    "# They are loaded at runtime by the launch file\n",
    "Alpha = rospy.get_param(\"/cartpole_v0/alpha\")\n",
    "Epsilon = rospy.get_param(\"/cartpole_v0/epsilon\")\n",
    "Gamma = rospy.get_param(\"/cartpole_v0/gamma\")\n",
    "epsilon_discount = rospy.get_param(\"/cartpole_v0/epsilon_discount\")\n",
    "nepisodes = rospy.get_param(\"/cartpole_v0/nepisodes\")\n",
    "nsteps = rospy.get_param(\"/cartpole_v0/nsteps\")\n",
    "\n",
    "running_step = rospy.get_param(\"/cartpole_v0/running_step\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to change the package **cartpole_v0_training** to our own package; in this case, we will pick **my_moving_cube_pkg**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, change the namespace from **cartpole_v0** to **moving_cube**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    # Set the logging system\n",
    "    rospack = rospkg.RosPack()\n",
    "    pkg_path = rospack.get_path('my_moving_cube_pkg')\n",
    "    outdir = pkg_path + '/training_results'\n",
    "    env = wrappers.Monitor(env, outdir, force=True) \n",
    "    rospy.loginfo ( \"Monitor Wrapper started\")\n",
    "\n",
    "    last_time_steps = numpy.ndarray(0)\n",
    "\n",
    "    # Loads parameters from the ROS param server\n",
    "    # Parameters are stored in a yaml file inside the config directory\n",
    "    # They are loaded at runtime by the launch file\n",
    "    Alpha = rospy.get_param(\"/moving_cube/alpha\")\n",
    "    Epsilon = rospy.get_param(\"/moving_cube/epsilon\")\n",
    "    Gamma = rospy.get_param(\"/moving_cube/gamma\")\n",
    "    epsilon_discount = rospy.get_param(\"/moving_cube/epsilon_discount\")\n",
    "    nepisodes = rospy.get_param(\"/moving_cube/nepisodes\")\n",
    "    nsteps = rospy.get_param(\"/moving_cube/nsteps\")\n",
    "\n",
    "    running_step = rospy.get_param(\"/moving_cube/running_step\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4 FilterObservations for Simpler States Generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We sometimes want to filter the observations given, so that the reinforcement algorithms go faster. We have to add the following lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize the environment and get first state of the robot\n",
    "observation = env.reset()\n",
    "state = ''.join(map(str, observation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turn it into this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize the environment and get first state of the robot\n",
    "observation = env.reset()\n",
    "simplified_observations = convert_obs_to_state(observation)\n",
    "state = ''.join(map(str, simplified_observations))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And also this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nextState = ''.join(map(str, observation))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turn it into this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "simplified_observations = convert_obs_to_state(observation)\n",
    "nextState = ''.join(map(str, simplified_observations))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and define this **convert_obs_to_state** method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_obs_to_state(observations):\n",
    "    \"\"\"\n",
    "    Converts the observations used for reward and so on to the essentials for the robot state\n",
    "    In this case, we only need the orientation of the cube and the speed of the disc.\n",
    "    The distance doesn't condition at all the actions\n",
    "    \"\"\"\n",
    "    disk_roll_vel = observations[0]\n",
    "    # roll_angle = observations[2]\n",
    "    y_linear_speed = observations[4]\n",
    "    yaw_angle = observations[5]\n",
    "\n",
    "    state_converted = [disk_roll_vel, y_linear_speed, yaw_angle]\n",
    "\n",
    "    return state_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the training script is done. You now have to create a launch and a param_config file to start the training system. Here are the files:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**my_one_disk_walk_openai_params.yaml**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "moving_cube: #namespace\n",
    "\n",
    "    running_step: 0.04 # amount of time the control will be executed\n",
    "    pos_step: 0.016     # increment in position for each command\n",
    "    \n",
    "    #qlearn parameters\n",
    "    alpha: 0.1\n",
    "    gamma: 0.7\n",
    "    epsilon: 0.9\n",
    "    epsilon_discount: 0.999\n",
    "    nepisodes: 500\n",
    "    nsteps: 1000\n",
    "    number_splits: 10 #set to change the number of state splits for the continuous problem and also the number of env_variable splits\n",
    "\n",
    "    running_step: 0.06 # Time for each step\n",
    "    wait_time: 0.1 # Time to wait in the reset phases\n",
    "\n",
    "    n_actions: 5 # We have 3 actions\n",
    "\n",
    "    speed_step: 1.0 # Time to wait in the reset phases\n",
    "\n",
    "    init_roll_vel: 0.0 # Initial speed of the Roll Disk\n",
    "\n",
    "    roll_speed_fixed_value: 100.0 # Speed at which it will move forwards or backwards\n",
    "    roll_speed_increment_value: 10.0 # Increment that could be done in each step\n",
    "\n",
    "    max_distance: 2.0 # Maximum distance allowed for the RobotCube\n",
    "    max_pitch_angle: 0.2 # Maximum Angle radians in Pitch that we allow before terminating episode\n",
    "    max_yaw_angle: 0.1 # Maximum yaw angle deviation, after that it starts getting negative rewards\n",
    "    max_y_linear_speed: 100 # Maximum speed in y axis\n",
    "\n",
    "    init_cube_pose:\n",
    "      x: 0.0\n",
    "      y: 0.0\n",
    "      z: 0.0\n",
    "\n",
    "    end_episode_points: 1000 # Points given when ending an episode\n",
    "\n",
    "    move_distance_reward_weight: 1000.0 # Multiplier for the moved distance reward, Ex: inc_d = 0.1 --> 100points\n",
    "    y_linear_speed_reward_weight: 1000.0 # Multiplier for moving fast on the y Axis\n",
    "    y_axis_angle_reward_weight: 1000.0 # Multiplier of angle of yaw, to keep it straight\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**END my_one_disk_walk_openai_params.yaml**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**start_training_cube.launch**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<launch>\n",
    "    <rosparam command=\"load\" file=\"$(find my_moving_cube_pkg)/config/my_one_disk_walk_openai_params.yaml\" />\n",
    "    <!-- Launch the training system -->\n",
    "    <node pkg=\"my_moving_cube_pkg\" name=\"movingcube_gym\" type=\"start_training.py\" output=\"screen\"/>\n",
    "</launch>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**END start_training_cube.launch**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**start_training.py**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#!/usr/bin/env python\n",
    "\n",
    "import gym\n",
    "import time\n",
    "import numpy\n",
    "import random\n",
    "import time\n",
    "import qlearn\n",
    "from gym import wrappers\n",
    "\n",
    "# ROS packages required\n",
    "import rospy\n",
    "import rospkg\n",
    "\n",
    "# import our training environment\n",
    "#from openai_ros.task_envs.cartpole_stay_up import stay_up\n",
    "# import our training environment\n",
    "import my_one_disk_walk\n",
    "\n",
    "\n",
    "def convert_obs_to_state(observations):\n",
    "    \"\"\"\n",
    "    Converts the observations used for reward and so on to the essentials for the robot state\n",
    "    In this case, we only need the orientation of the cube and the speed of the disc.\n",
    "    The distance doesn't condition at all the actions\n",
    "    \"\"\"\n",
    "    disk_roll_vel = observations[0]\n",
    "    # roll_angle = observations[2]\n",
    "    y_linear_speed = observations[4]\n",
    "    yaw_angle = observations[5]\n",
    "\n",
    "    state_converted = [disk_roll_vel, y_linear_speed, yaw_angle]\n",
    "\n",
    "    return state_converted\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    rospy.init_node('cartpole_gym', anonymous=True, log_level=rospy.WARN)\n",
    "\n",
    "    # Create the Gym environment\n",
    "    #env = gym.make('CartPoleStayUp-v0')\n",
    "    env = gym.make('MyMovingCubeOneDiskWalkEnv-v0')\n",
    "    rospy.loginfo ( \"Gym environment done\")\n",
    "        \n",
    "    # Set the logging system\n",
    "    rospack = rospkg.RosPack()\n",
    "    pkg_path = rospack.get_path('my_moving_cube_pkg')\n",
    "    outdir = pkg_path + '/training_results'\n",
    "    env = wrappers.Monitor(env, outdir, force=True) \n",
    "    rospy.loginfo ( \"Monitor Wrapper started\")\n",
    "\n",
    "    last_time_steps = numpy.ndarray(0)\n",
    "\n",
    "    # Loads parameters from the ROS param server\n",
    "    # Parameters are stored in a yaml file inside the config directory\n",
    "    # They are loaded at runtime by the launch file\n",
    "    Alpha = rospy.get_param(\"/moving_cube/alpha\")\n",
    "    Epsilon = rospy.get_param(\"/moving_cube/epsilon\")\n",
    "    Gamma = rospy.get_param(\"/moving_cube/gamma\")\n",
    "    epsilon_discount = rospy.get_param(\"/moving_cube/epsilon_discount\")\n",
    "    nepisodes = rospy.get_param(\"/moving_cube/nepisodes\")\n",
    "    nsteps = rospy.get_param(\"/moving_cube/nsteps\")\n",
    "\n",
    "    running_step = rospy.get_param(\"/moving_cube/running_step\")\n",
    "\n",
    "    # Initialises the algorithm that we are going to use for learning\n",
    "    qlearn = qlearn.QLearn(actions=range(env.action_space.n),\n",
    "                    alpha=Alpha, gamma=Gamma, epsilon=Epsilon)\n",
    "    initial_epsilon = qlearn.epsilon\n",
    "\n",
    "    start_time = time.time()\n",
    "    highest_reward = 0\n",
    "\n",
    "    # Starts the main training loop: the one about the episodes to do\n",
    "    for x in range(nepisodes):\n",
    "        rospy.logdebug(\"############### START EPISODE => \" + str(x))\n",
    "        \n",
    "        cumulated_reward = 0  \n",
    "        done = False\n",
    "        if qlearn.epsilon > 0.05:\n",
    "            qlearn.epsilon *= epsilon_discount\n",
    "        \n",
    "        # Initialize the environment and get first state of the robot\n",
    "        observation = env.reset()\n",
    "        simplified_observations = convert_obs_to_state(observation)\n",
    "        state = ''.join(map(str, simplified_observations))\n",
    "        \n",
    "        # Show on screen the actual situation of the robot\n",
    "        # for each episode, we test the robot for nsteps\n",
    "        for i in range(nsteps):\n",
    "            \n",
    "            rospy.loginfo(\"############### Start Step => \"+str(i))\n",
    "            # Pick an action based on the current state\n",
    "            action = qlearn.chooseAction(state)\n",
    "            rospy.loginfo (\"Next action is: %d\", action)\n",
    "            # Execute the action in the environment and get feedback\n",
    "            observation, reward, done, info = env.step(action)\n",
    "            rospy.loginfo(str(observation) + \" \" + str(reward))\n",
    "            cumulated_reward += reward\n",
    "            if highest_reward < cumulated_reward:\n",
    "                highest_reward = cumulated_reward\n",
    "\n",
    "            simplified_observations = convert_obs_to_state(observation)\n",
    "            nextState = ''.join(map(str, simplified_observations))\n",
    "\n",
    "            # Make the algorithm learn based on the results\n",
    "            #rospy.logwarn(\"############### State we were => \" + str(state))\n",
    "            #rospy.logwarn(\"############### Action that we took => \" + str(action))\n",
    "            #rospy.logwarn(\"############### Reward that action gave => \" + str(reward))\n",
    "            #rospy.logwarn(\"############### State in which we will start next step => \" + str(nextState))\n",
    "            qlearn.learn(state, action, reward, nextState)\n",
    "\n",
    "            if not(done):\n",
    "                state = nextState\n",
    "            else:\n",
    "                rospy.loginfo (\"DONE\")\n",
    "                last_time_steps = numpy.append(last_time_steps, [int(i + 1)])\n",
    "                break\n",
    "            rospy.loginfo(\"############### END Step =>\" + str(i))\n",
    "            #raw_input(\"Next Step...PRESS KEY\")\n",
    "            #rospy.sleep(2.0)\n",
    "        m, s = divmod(int(time.time() - start_time), 60)\n",
    "        h, m = divmod(m, 60)\n",
    "        rospy.logwarn ( (\"EP: \"+str(x+1)+\" - [alpha: \"+str(round(qlearn.alpha,2))+\" - gamma: \"+str(round(qlearn.gamma,2))+\" - epsilon: \"+str(round(qlearn.epsilon,2))+\"] - Reward: \"+str(cumulated_reward)+\"     Time: %d:%02d:%02d\" % (h, m, s)))\n",
    "\n",
    "    \n",
    "    rospy.loginfo ( (\"\\n|\"+str(nepisodes)+\"|\"+str(qlearn.alpha)+\"|\"+str(qlearn.gamma)+\"|\"+str(initial_epsilon)+\"*\"+str(epsilon_discount)+\"|\"+str(highest_reward)+\"| PICTURE |\"))\n",
    "\n",
    "    l = last_time_steps.tolist()\n",
    "    l.sort()\n",
    "\n",
    "    #print(\"Parameters: a=\"+str)\n",
    "    rospy.loginfo(\"Overall score: {:0.2f}\".format(last_time_steps.mean()))\n",
    "    rospy.loginfo(\"Best 100 score: {:0.2f}\".format(reduce(lambda x, y: x + y, l[-100:]) / len(l[-100:])))\n",
    "\n",
    "    env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**END start_training.py**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the Qlearn python module, just in case you don't have it:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**qlearn.py**</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "Q-learning approach for different RL problems\n",
    "as part of the basic series on reinforcement learning @\n",
    "https://github.com/vmayoral/basic_reinforcement_learning\n",
    " \n",
    "Inspired by https://gym.openai.com/evaluations/eval_kWknKOkPQ7izrixdhriurA\n",
    " \n",
    "        @author: Victor Mayoral Vilches <victor@erlerobotics.com>\n",
    "'''\n",
    "import random\n",
    "\n",
    "class QLearn:\n",
    "    def __init__(self, actions, epsilon, alpha, gamma):\n",
    "        self.q = {}\n",
    "        self.epsilon = epsilon  # exploration constant\n",
    "        self.alpha = alpha      # discount constant\n",
    "        self.gamma = gamma      # discount factor\n",
    "        self.actions = actions\n",
    "\n",
    "    def getQ(self, state, action):\n",
    "        return self.q.get((state, action), 0.0)\n",
    "\n",
    "    def learnQ(self, state, action, reward, value):\n",
    "        '''\n",
    "        Q-learning:\n",
    "            Q(s, a) += alpha * (reward(s,a) + max(Q(s') - Q(s,a))            \n",
    "        '''\n",
    "        oldv = self.q.get((state, action), None)\n",
    "        if oldv is None:\n",
    "            self.q[(state, action)] = reward\n",
    "        else:\n",
    "            self.q[(state, action)] = oldv + self.alpha * (value - oldv)\n",
    "\n",
    "    def chooseAction(self, state, return_q=False):\n",
    "        q = [self.getQ(state, a) for a in self.actions]\n",
    "        maxQ = max(q)\n",
    "\n",
    "        if random.random() < self.epsilon:\n",
    "            minQ = min(q); mag = max(abs(minQ), abs(maxQ))\n",
    "            # add random values to all the actions, recalculate maxQ\n",
    "            q = [q[i] + random.random() * mag - .5 * mag for i in range(len(self.actions))] \n",
    "            maxQ = max(q)\n",
    "\n",
    "        count = q.count(maxQ)\n",
    "        # In case there're several state-action max values \n",
    "        # we select a random one among them\n",
    "        if count > 1:\n",
    "            best = [i for i in range(len(self.actions)) if q[i] == maxQ]\n",
    "            i = random.choice(best)\n",
    "        else:\n",
    "            i = q.index(maxQ)\n",
    "\n",
    "        action = self.actions[i]        \n",
    "        if return_q: # if they want it, give it!\n",
    "            return action, q\n",
    "        return action\n",
    "\n",
    "    def learn(self, state1, action1, reward, state2):\n",
    "        maxqnew = max([self.getQ(state2, a) for a in self.actions])\n",
    "        self.learnQ(state1, action1, reward, reward + self.gamma*maxqnew)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#407EAF;color:white;\">**END qlearn.py**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see something like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/movingcube_learning.gif\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#FF5E00;color:white;\">**END TEST new Environment**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#FF5E00;color:white;\">**EXTRA exercise U2.1**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a challenge, how would we change the direction in which we learn to move?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:#FF5E00;color:white;\">**END EXTRA exercise U2.1**</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">Solution Exercise U2.1</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please try to do it by yourself unless you get stuck or need some inspiration. You will learn much more if you fight for each exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/robotignite_logo_text.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, the answer is simple: We go to the **MyMovingCubeOneDiskWalkEnv** and change the way in which we assign the rewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_linear_speed = observations[4]\n",
    "rospy.logdebug(\"y_linear_speed=\" + str(y_linear_speed))\n",
    "reward_y_axis_speed = y_linear_speed * self.y_linear_speed_reward_weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have to invert the sign, although we don't even need to access the code. We just have to change the value of the self.y_linear_speed_reward_weight, which is imported from the yaml file that defines the parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_linear_speed_reward_weight: 1000.0 # Multiplier for moving fast in the y Axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_linear_speed_reward_weight: -1000.0 # Multiplier for moving fast in the y Axis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/movingcube_learning_backwards2.gif\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:green;color:white;\">Solution Exercise U2.1</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NEXT STEP: How to change the learning algorithm"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}