Value 1.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python_defaultSpec_1600767430371",
   "display_name": "Python 3.7.8 64-bit"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Preprocess the data"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import BytesIO\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd  \n",
    "\n",
    "from sklearn.metrics import brier_score_loss, roc_auc_score \n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "import socceraction.vaep.features as features\n",
    "import socceraction.vaep.labels as labels\n",
    "\n",
    "from socceraction.spadl.wyscout import convert_to_spadl\n",
    "from socceraction.vaep.formula import value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore', category=pd.io.pytables.PerformanceWarning)"
   ]
  },
  {
   "source": [
    "## Preprocess the Wyscout data"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The `read_json_file` function reads and returns the content of a given JSON file. The function handles the encoding of special characters (e.g., accents in names of players and teams) that the `pd.read_json` function cannot handle properly."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_json_file(filename):\n",
    "    with open(filename, 'rb') as json_file:\n",
    "        return BytesIO(json_file.read()).getvalue().decode('unicode_escape')"
   ]
  },
  {
   "source": [
    "### Teams"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cells read the `teams.json` file into a `DataFrame` object and store that object in the `wyscout.h5` HDF5 file under the key `teams`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_teams = read_json_file('Wyscout/teams.json')\n",
    "df_teams = pd.read_json(json_teams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "                           city                 name  wyId  \\\n0           Newcastle upon Tyne     Newcastle United  1613   \n1                          Vigo        Celta de Vigo   692   \n2                     Barcelona             Espanyol   691   \n3               Vitoria-Gasteiz     Deportivo Alavés   696   \n4                      Valencia              Levante   695   \n5                        Troyes               Troyes  3795   \n6               Getafe (Madrid)               Getafe   698   \n7               Mönchengladbach  Borussia M'gladbach  2454   \n8  Huddersfield, West Yorkshire    Huddersfield Town  1673   \n9                        Bilbao        Athletic Club   678   \n\n                               officialName  \\\n0                       Newcastle United FC   \n1                   Real Club Celta de Vigo   \n2              Reial Club Deportiu Espanyol   \n3                          Deportivo Alavés   \n4                                Levante UD   \n5  Espérance Sportive Troyes Aube Champagne   \n6                     Getafe Club de Fútbol   \n7              Borussia VfL Mönchengladbach   \n8                      Huddersfield Town FC   \n9                      Athletic Club Bilbao   \n\n                                                area  type  \n0  {'name': 'England', 'id': '0', 'alpha3code': '...  club  \n1  {'name': 'Spain', 'id': '724', 'alpha3code': '...  club  \n2  {'name': 'Spain', 'id': '724', 'alpha3code': '...  club  \n3  {'name': 'Spain', 'id': '724', 'alpha3code': '...  club  \n4  {'name': 'Spain', 'id': '724', 'alpha3code': '...  club  \n5  {'name': 'France', 'id': '250', 'alpha3code': ...  club  \n6  {'name': 'Spain', 'id': '724', 'alpha3code': '...  club  \n7  {'name': 'Germany', 'id': '276', 'alpha3code':...  club  \n8  {'name': 'England', 'id': '0', 'alpha3code': '...  club  \n9  {'name': 'Spain', 'id': '724', 'alpha3code': '...  club  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>city</th>\n      <th>name</th>\n      <th>wyId</th>\n      <th>officialName</th>\n      <th>area</th>\n      <th>type</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Newcastle upon Tyne</td>\n      <td>Newcastle United</td>\n      <td>1613</td>\n      <td>Newcastle United FC</td>\n      <td>{'name': 'England', 'id': '0', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Vigo</td>\n      <td>Celta de Vigo</td>\n      <td>692</td>\n      <td>Real Club Celta de Vigo</td>\n      <td>{'name': 'Spain', 'id': '724', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Barcelona</td>\n      <td>Espanyol</td>\n      <td>691</td>\n      <td>Reial Club Deportiu Espanyol</td>\n      <td>{'name': 'Spain', 'id': '724', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Vitoria-Gasteiz</td>\n      <td>Deportivo Alavés</td>\n      <td>696</td>\n      <td>Deportivo Alavés</td>\n      <td>{'name': 'Spain', 'id': '724', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Valencia</td>\n      <td>Levante</td>\n      <td>695</td>\n      <td>Levante UD</td>\n      <td>{'name': 'Spain', 'id': '724', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Troyes</td>\n      <td>Troyes</td>\n      <td>3795</td>\n      <td>Espérance Sportive Troyes Aube Champagne</td>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Getafe (Madrid)</td>\n      <td>Getafe</td>\n      <td>698</td>\n      <td>Getafe Club de Fútbol</td>\n      <td>{'name': 'Spain', 'id': '724', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Mönchengladbach</td>\n      <td>Borussia M'gladbach</td>\n      <td>2454</td>\n      <td>Borussia VfL Mönchengladbach</td>\n      <td>{'name': 'Germany', 'id': '276', 'alpha3code':...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Huddersfield, West Yorkshire</td>\n      <td>Huddersfield Town</td>\n      <td>1673</td>\n      <td>Huddersfield Town FC</td>\n      <td>{'name': 'England', 'id': '0', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Bilbao</td>\n      <td>Athletic Club</td>\n      <td>678</td>\n      <td>Athletic Club Bilbao</td>\n      <td>{'name': 'Spain', 'id': '724', 'alpha3code': '...</td>\n      <td>club</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 6
    }
   ],
   "source": [
    "df_teams.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_teams.to_hdf('wyscout.h5', key='teams', mode='w')"
   ]
  },
  {
   "source": [
    "### Players"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cells read the `players.json` file into a `DataFrame` object and store that object in the `wyscout.h5` HDF5 file under the key `players`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "json_players = read_json_file('Wyscout/players.json')\n",
    "df_players = pd.read_json(json_players)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "                                        passportArea  weight  \\\n0  {'name': 'Turkey', 'id': '792', 'alpha3code': ...      78   \n1  {'name': 'Senegal', 'id': '686', 'alpha3code':...      73   \n2  {'name': 'France', 'id': '250', 'alpha3code': ...      72   \n3  {'name': 'Senegal', 'id': '686', 'alpha3code':...      82   \n4  {'name': 'France', 'id': '250', 'alpha3code': ...      84   \n5  {'name': 'Netherlands', 'id': '528', 'alpha3co...      83   \n6  {'name': 'Belgium', 'id': '56', 'alpha3code': ...      91   \n7  {'name': 'Belgium', 'id': '56', 'alpha3code': ...      88   \n8  {'name': 'France', 'id': '250', 'alpha3code': ...      74   \n9  {'name': 'Denmark', 'id': '208', 'alpha3code':...      76   \n\n           firstName middleName           lastName currentTeamId   birthDate  \\\n0              Harun                         Tekin          4502  1989-06-17   \n1             Malang                          Sarr          3775  1999-01-23   \n2               Over                      Mandanda          3772  1998-10-26   \n3  Alfred John Momar                       N'Diaye           683  1990-03-06   \n4           Ibrahima                        Konaté          2975  1999-05-25   \n5             Jasper                     Cillessen           676  1989-04-22   \n6               Toby                  Alderweireld          1624  1989-03-02   \n7                Jan                    Vertonghen          1624  1987-04-24   \n8          Alexander                         Djiku          3783  1994-08-09   \n9         Christian              Dannemann Eriksen          1624  1992-02-14   \n\n   height                                               role  \\\n0     187  {'code2': 'GK', 'code3': 'GKP', 'name': 'Goalk...   \n1     182  {'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...   \n2     176  {'code2': 'GK', 'code3': 'GKP', 'name': 'Goalk...   \n3     187  {'code2': 'MD', 'code3': 'MID', 'name': 'Midfi...   \n4     192  {'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...   \n5     185  {'code2': 'GK', 'code3': 'GKP', 'name': 'Goalk...   \n6     187  {'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...   \n7     189  {'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...   \n8     182  {'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...   \n9     180  {'code2': 'MD', 'code3': 'MID', 'name': 'Midfi...   \n\n                                           birthArea    wyId   foot  \\\n0  {'name': 'Turkey', 'id': '792', 'alpha3code': ...   32777  right   \n1  {'name': 'France', 'id': '250', 'alpha3code': ...  393228   left   \n2  {'name': 'France', 'id': '250', 'alpha3code': ...  393230          \n3  {'name': 'France', 'id': '250', 'alpha3code': ...   32793  right   \n4  {'name': 'France', 'id': '250', 'alpha3code': ...  393247  right   \n5  {'name': 'Netherlands', 'id': '528', 'alpha3co...      33  right   \n6  {'name': 'Belgium', 'id': '56', 'alpha3code': ...      36  right   \n7  {'name': 'Belgium', 'id': '56', 'alpha3code': ...      48   left   \n8  {'name': 'France', 'id': '250', 'alpha3code': ...  229427  right   \n9  {'name': 'Denmark', 'id': '208', 'alpha3code':...      54  right   \n\n         shortName currentNationalTeamId  \n0         H. Tekin                  4687  \n1          M. Sarr                  4423  \n2      O. Mandanda                  null  \n3       A. N'Diaye                 19314  \n4        I. Konaté                  null  \n5     J. Cillessen                   664  \n6  T. Alderweireld                  5629  \n7    J. Vertonghen                  5629  \n8         A. Djiku                  null  \n9       C. Eriksen                  7712  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>passportArea</th>\n      <th>weight</th>\n      <th>firstName</th>\n      <th>middleName</th>\n      <th>lastName</th>\n      <th>currentTeamId</th>\n      <th>birthDate</th>\n      <th>height</th>\n      <th>role</th>\n      <th>birthArea</th>\n      <th>wyId</th>\n      <th>foot</th>\n      <th>shortName</th>\n      <th>currentNationalTeamId</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>{'name': 'Turkey', 'id': '792', 'alpha3code': ...</td>\n      <td>78</td>\n      <td>Harun</td>\n      <td></td>\n      <td>Tekin</td>\n      <td>4502</td>\n      <td>1989-06-17</td>\n      <td>187</td>\n      <td>{'code2': 'GK', 'code3': 'GKP', 'name': 'Goalk...</td>\n      <td>{'name': 'Turkey', 'id': '792', 'alpha3code': ...</td>\n      <td>32777</td>\n      <td>right</td>\n      <td>H. Tekin</td>\n      <td>4687</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>{'name': 'Senegal', 'id': '686', 'alpha3code':...</td>\n      <td>73</td>\n      <td>Malang</td>\n      <td></td>\n      <td>Sarr</td>\n      <td>3775</td>\n      <td>1999-01-23</td>\n      <td>182</td>\n      <td>{'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...</td>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>393228</td>\n      <td>left</td>\n      <td>M. Sarr</td>\n      <td>4423</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>72</td>\n      <td>Over</td>\n      <td></td>\n      <td>Mandanda</td>\n      <td>3772</td>\n      <td>1998-10-26</td>\n      <td>176</td>\n      <td>{'code2': 'GK', 'code3': 'GKP', 'name': 'Goalk...</td>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>393230</td>\n      <td></td>\n      <td>O. Mandanda</td>\n      <td>null</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>{'name': 'Senegal', 'id': '686', 'alpha3code':...</td>\n      <td>82</td>\n      <td>Alfred John Momar</td>\n      <td></td>\n      <td>N'Diaye</td>\n      <td>683</td>\n      <td>1990-03-06</td>\n      <td>187</td>\n      <td>{'code2': 'MD', 'code3': 'MID', 'name': 'Midfi...</td>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>32793</td>\n      <td>right</td>\n      <td>A. N'Diaye</td>\n      <td>19314</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>84</td>\n      <td>Ibrahima</td>\n      <td></td>\n      <td>Konaté</td>\n      <td>2975</td>\n      <td>1999-05-25</td>\n      <td>192</td>\n      <td>{'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...</td>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>393247</td>\n      <td>right</td>\n      <td>I. Konaté</td>\n      <td>null</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>{'name': 'Netherlands', 'id': '528', 'alpha3co...</td>\n      <td>83</td>\n      <td>Jasper</td>\n      <td></td>\n      <td>Cillessen</td>\n      <td>676</td>\n      <td>1989-04-22</td>\n      <td>185</td>\n      <td>{'code2': 'GK', 'code3': 'GKP', 'name': 'Goalk...</td>\n      <td>{'name': 'Netherlands', 'id': '528', 'alpha3co...</td>\n      <td>33</td>\n      <td>right</td>\n      <td>J. Cillessen</td>\n      <td>664</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>{'name': 'Belgium', 'id': '56', 'alpha3code': ...</td>\n      <td>91</td>\n      <td>Toby</td>\n      <td></td>\n      <td>Alderweireld</td>\n      <td>1624</td>\n      <td>1989-03-02</td>\n      <td>187</td>\n      <td>{'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...</td>\n      <td>{'name': 'Belgium', 'id': '56', 'alpha3code': ...</td>\n      <td>36</td>\n      <td>right</td>\n      <td>T. Alderweireld</td>\n      <td>5629</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>{'name': 'Belgium', 'id': '56', 'alpha3code': ...</td>\n      <td>88</td>\n      <td>Jan</td>\n      <td></td>\n      <td>Vertonghen</td>\n      <td>1624</td>\n      <td>1987-04-24</td>\n      <td>189</td>\n      <td>{'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...</td>\n      <td>{'name': 'Belgium', 'id': '56', 'alpha3code': ...</td>\n      <td>48</td>\n      <td>left</td>\n      <td>J. Vertonghen</td>\n      <td>5629</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>74</td>\n      <td>Alexander</td>\n      <td></td>\n      <td>Djiku</td>\n      <td>3783</td>\n      <td>1994-08-09</td>\n      <td>182</td>\n      <td>{'code2': 'DF', 'code3': 'DEF', 'name': 'Defen...</td>\n      <td>{'name': 'France', 'id': '250', 'alpha3code': ...</td>\n      <td>229427</td>\n      <td>right</td>\n      <td>A. Djiku</td>\n      <td>null</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>{'name': 'Denmark', 'id': '208', 'alpha3code':...</td>\n      <td>76</td>\n      <td>Christian</td>\n      <td></td>\n      <td>Dannemann Eriksen</td>\n      <td>1624</td>\n      <td>1992-02-14</td>\n      <td>180</td>\n      <td>{'code2': 'MD', 'code3': 'MID', 'name': 'Midfi...</td>\n      <td>{'name': 'Denmark', 'id': '208', 'alpha3code':...</td>\n      <td>54</td>\n      <td>right</td>\n      <td>C. Eriksen</td>\n      <td>7712</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "df_players.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_players.to_hdf('wyscout.h5', key='players', mode='a')"
   ]
  },
  {
   "source": [
    "### Matches"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell lists the competitions to be included in the dataset. Uncomment the competitions that you want to include in your dataset."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "competitions = [\n",
    "   'England',\n",
    "#     'France',\n",
    "#     'Germany',\n",
    "#     'Italy',\n",
    "#     'Spain',\n",
    "#   'European Championship',\n",
    "#     'World Cup'\n",
    "]"
   ]
  },
  {
   "source": [
    "The following cells read the `matches.json` files for the selected competitions into a `DataFrame` object and store that object in the `wyscout.h5` HDF5 file under the key `matches`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_matches = []\n",
    "for competition in competitions:\n",
    "    competition_name = competition.replace(' ', '_')\n",
    "    file_matches = f'Wyscout/matches/matches_{competition_name}.json'\n",
    "    json_matches = read_json_file(file_matches)\n",
    "    df_matches = pd.read_json(json_matches)\n",
    "    dfs_matches.append(df_matches)\n",
    "df_matches = pd.concat(dfs_matches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   status  roundId  gameweek  \\\n0  Played  4405654        38   \n1  Played  4405654        38   \n2  Played  4405654        38   \n3  Played  4405654        38   \n4  Played  4405654        38   \n5  Played  4405654        38   \n6  Played  4405654        38   \n7  Played  4405654        38   \n8  Played  4405654        38   \n9  Played  4405654        38   \n\n                                           teamsData  seasonId  \\\n0  {'1646': {'scoreET': 0, 'coachId': 8880, 'side...    181150   \n1  {'1628': {'scoreET': 0, 'coachId': 8357, 'side...    181150   \n2  {'1609': {'scoreET': 0, 'coachId': 7845, 'side...    181150   \n3  {'1651': {'scoreET': 0, 'coachId': 8093, 'side...    181150   \n4  {'1644': {'scoreET': 0, 'coachId': 93112, 'sid...    181150   \n5  {'1613': {'scoreET': 0, 'coachId': 210700, 'si...    181150   \n6  {'1625': {'scoreET': 0, 'coachId': 267136, 'si...    181150   \n7  {'10531': {'scoreET': 0, 'coachId': 32573, 'si...    181150   \n8  {'1631': {'scoreET': 0, 'coachId': 209010, 'si...    181150   \n9  {'1623': {'scoreET': 0, 'coachId': 8541, 'side...    181150   \n\n               dateutc  winner                     venue     wyId  \\\n0  2018-05-13 14:00:00    1659                 Turf Moor  2500089   \n1  2018-05-13 14:00:00    1628             Selhurst Park  2500090   \n2  2018-05-13 14:00:00    1609  The John Smith's Stadium  2500091   \n3  2018-05-13 14:00:00    1612                   Anfield  2500092   \n4  2018-05-13 14:00:00    1611              Old Trafford  2500093   \n5  2018-05-13 14:00:00    1613           St. James' Park  2500094   \n6  2018-05-13 14:00:00    1625        St. Mary's Stadium  2500095   \n7  2018-05-13 14:00:00    1639           Liberty Stadium  2500096   \n8  2018-05-13 14:00:00    1624           Wembley Stadium  2500097   \n9  2018-05-13 14:00:00    1633            London Stadium  2500098   \n\n                                          label  \\\n0              Burnley - AFC Bournemouth, 1 - 2   \n1  Crystal Palace - West Bromwich Albion, 2 - 0   \n2            Huddersfield Town - Arsenal, 0 - 1   \n3     Liverpool - Brighton & Hove Albion, 4 - 0   \n4            Manchester United - Watford, 1 - 0   \n5             Newcastle United - Chelsea, 3 - 0   \n6          Southampton - Manchester City, 0 - 1   \n7              Swansea City - Stoke City, 1 - 2   \n8     Tottenham Hotspur - Leicester City, 5 - 4   \n9              West Ham United - Everton, 3 - 1   \n\n                               date  \\\n0  May 13, 2018 at 4:00:00 PM GMT+2   \n1  May 13, 2018 at 4:00:00 PM GMT+2   \n2  May 13, 2018 at 4:00:00 PM GMT+2   \n3  May 13, 2018 at 4:00:00 PM GMT+2   \n4  May 13, 2018 at 4:00:00 PM GMT+2   \n5  May 13, 2018 at 4:00:00 PM GMT+2   \n6  May 13, 2018 at 4:00:00 PM GMT+2   \n7  May 13, 2018 at 4:00:00 PM GMT+2   \n8  May 13, 2018 at 4:00:00 PM GMT+2   \n9  May 13, 2018 at 4:00:00 PM GMT+2   \n\n                                            referees duration  competitionId  \n0  [{'refereeId': 385705, 'role': 'referee'}, {'r...  Regular            364  \n1  [{'refereeId': 381851, 'role': 'referee'}, {'r...  Regular            364  \n2  [{'refereeId': 384965, 'role': 'referee'}, {'r...  Regular            364  \n3  [{'refereeId': 385704, 'role': 'referee'}, {'r...  Regular            364  \n4  [{'refereeId': 381853, 'role': 'referee'}, {'r...  Regular            364  \n5  [{'refereeId': 384888, 'role': 'referee'}, {'r...  Regular            364  \n6  [{'refereeId': 385911, 'role': 'referee'}, {'r...  Regular            364  \n7  [{'refereeId': 378952, 'role': 'referee'}, {'r...  Regular            364  \n8  [{'refereeId': 378951, 'role': 'referee'}, {'r...  Regular            364  \n9  [{'refereeId': 408156, 'role': 'referee'}, {'r...  Regular            364  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>status</th>\n      <th>roundId</th>\n      <th>gameweek</th>\n      <th>teamsData</th>\n      <th>seasonId</th>\n      <th>dateutc</th>\n      <th>winner</th>\n      <th>venue</th>\n      <th>wyId</th>\n      <th>label</th>\n      <th>date</th>\n      <th>referees</th>\n      <th>duration</th>\n      <th>competitionId</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1646': {'scoreET': 0, 'coachId': 8880, 'side...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1659</td>\n      <td>Turf Moor</td>\n      <td>2500089</td>\n      <td>Burnley - AFC Bournemouth, 1 - 2</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 385705, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1628': {'scoreET': 0, 'coachId': 8357, 'side...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1628</td>\n      <td>Selhurst Park</td>\n      <td>2500090</td>\n      <td>Crystal Palace - West Bromwich Albion, 2 - 0</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 381851, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1609': {'scoreET': 0, 'coachId': 7845, 'side...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1609</td>\n      <td>The John Smith's Stadium</td>\n      <td>2500091</td>\n      <td>Huddersfield Town - Arsenal, 0 - 1</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 384965, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1651': {'scoreET': 0, 'coachId': 8093, 'side...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1612</td>\n      <td>Anfield</td>\n      <td>2500092</td>\n      <td>Liverpool - Brighton &amp; Hove Albion, 4 - 0</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 385704, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1644': {'scoreET': 0, 'coachId': 93112, 'sid...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1611</td>\n      <td>Old Trafford</td>\n      <td>2500093</td>\n      <td>Manchester United - Watford, 1 - 0</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 381853, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1613': {'scoreET': 0, 'coachId': 210700, 'si...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1613</td>\n      <td>St. James' Park</td>\n      <td>2500094</td>\n      <td>Newcastle United - Chelsea, 3 - 0</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 384888, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1625': {'scoreET': 0, 'coachId': 267136, 'si...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1625</td>\n      <td>St. Mary's Stadium</td>\n      <td>2500095</td>\n      <td>Southampton - Manchester City, 0 - 1</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 385911, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'10531': {'scoreET': 0, 'coachId': 32573, 'si...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1639</td>\n      <td>Liberty Stadium</td>\n      <td>2500096</td>\n      <td>Swansea City - Stoke City, 1 - 2</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 378952, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1631': {'scoreET': 0, 'coachId': 209010, 'si...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1624</td>\n      <td>Wembley Stadium</td>\n      <td>2500097</td>\n      <td>Tottenham Hotspur - Leicester City, 5 - 4</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 378951, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Played</td>\n      <td>4405654</td>\n      <td>38</td>\n      <td>{'1623': {'scoreET': 0, 'coachId': 8541, 'side...</td>\n      <td>181150</td>\n      <td>2018-05-13 14:00:00</td>\n      <td>1633</td>\n      <td>London Stadium</td>\n      <td>2500098</td>\n      <td>West Ham United - Everton, 3 - 1</td>\n      <td>May 13, 2018 at 4:00:00 PM GMT+2</td>\n      <td>[{'refereeId': 408156, 'role': 'referee'}, {'r...</td>\n      <td>Regular</td>\n      <td>364</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 13
    }
   ],
   "source": [
    "df_matches.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_matches.to_hdf('wyscout.h5', key='matches', mode='a')"
   ]
  },
  {
   "source": [
    "### Events"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cells read the `events.json` files for the selected competitions into a `DataFrame` object and store that object in the `wyscout.h5` HDF5 file under the key `events/match_<match-id>`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "for competition in competitions:\n",
    "    competition_name = competition.replace(' ', '_')\n",
    "    file_events = f'Wyscout/events/events_{competition_name}.json'\n",
    "    json_events = read_json_file(file_events)\n",
    "    df_events = pd.read_json(json_events)\n",
    "    df_events_matches = df_events.groupby('matchId', as_index=False)\n",
    "    for match_id, df_events_match in df_events_matches:\n",
    "        df_events_match.to_hdf('wyscout.h5', key=f'events/match_{match_id}', mode='a')"
   ]
  },
  {
   "source": [
    "## Convert the Wyscout data to the SPADL representation"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell calls the `convert_to_spadl` function from the `socceraction` library to convert the `wyscout.h5` HDF5 file into the `spadl.h5` HDF5 file."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "...Inserting actiontypes\n...Inserting bodyparts\n...Inserting results\n...Converting games\n...Converting players\n...Converting teams\n...Generating player_games\n100%|██████████| 380/380 [01:34<00:00,  4.01game/s]\n  0%|          | 0/380 [00:00<?, ?game/s]...Converting events to actions\n100%|██████████| 380/380 [07:53<00:00,  1.25s/game]\n"
    }
   ],
   "source": [
    "convert_to_spadl('wyscout.h5', 'spadl.h5')"
   ]
  },
  {
   "source": [
    "# Value game states"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "This section generates features and labels for the game states, trains a predictive machine learning model for each label, and values the game states by applying the trained machine learning models.\n",
    "\n",
    "1. Generate the features to describe the game states;\n",
    "2. Generate the labels that capture the value of the game states;\n",
    "3. Compose a dataset by selecting a set of features and the labels of the game states;\n",
    "4. Train predictive machine learning models using the dataset;\n",
    "5. Value the game states using the trained predictive machine learning model.\n",
    "\n",
    "**Note:** The code in this section is based on the [2-compute-features-and-labels.ipynb](https://github.com/ML-KULeuven/socceraction/blob/master/public-notebooks/2-compute-features-and-labels.ipynb) and [3-estimate-scoring-and-conceding-probabilities.ipynb](https://github.com/ML-KULeuven/socceraction/blob/master/public-notebooks/3-estimate-scoring-and-conceding-probabilities.ipynb) notebooks in the `socceraction` repository."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_games = pd.read_hdf('spadl.h5', key='games')\n",
    "df_actiontypes = pd.read_hdf('spadl.h5', key='actiontypes')\n",
    "df_bodyparts = pd.read_hdf('spadl.h5', key='bodyparts')\n",
    "df_results = pd.read_hdf('spadl.h5', key='results')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_prev_actions = 3"
   ]
  },
  {
   "source": [
    "## Generate game state features"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell lists a number of *feature generators* from the `features` module in the `socceraction` library. Each function expects either a `DataFrame` object containing actions (i.e., individual actions) or a list of `DataFrame` objects containing consecutive actions (i.e., game states), and returns the corresponding *feature* for the individual action or game state."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "functions_features = [\n",
    "    features.actiontype_onehot,\n",
    "    features.bodypart_onehot,\n",
    "    features.result_onehot,\n",
    "    features.goalscore,\n",
    "    features.startlocation,\n",
    "    features.endlocation,\n",
    "    features.movement,\n",
    "    features.space_delta,\n",
    "    features.startpolar,\n",
    "    features.endpolar,\n",
    "    features.team,\n",
    "    features.time_delta\n",
    "]"
   ]
  },
  {
   "source": [
    "The following cell generates game states from consecutive actions in each game and computes the features for each game state.\n",
    "\n",
    "1. Obtain the actions for the game (i.e., `df_actions`) by looping through the games;\n",
    "2. Construct game states of a given length from the actions (i.e., `dfs_gamestates`);\n",
    "3. Compute the features for the constructed game states (i.e., `df_features`) by looping through the list of *feature generators*."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "d1e0d6e8dc4f4aa9b3190189ad6810bc"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_actions = pd.read_hdf('spadl.h5', key=f'actions/game_{game_id}')\n",
    "    df_actions = (df_actions\n",
    "        .merge(df_actiontypes, how='left')\n",
    "        .merge(df_results, how='left')\n",
    "        .merge(df_bodyparts, how='left')\n",
    "        .reset_index(drop=True)\n",
    "    )\n",
    "    \n",
    "    dfs_gamestates = features.gamestates(df_actions, nb_prev_actions=nb_prev_actions)\n",
    "    dfs_gamestates = features.play_left_to_right(dfs_gamestates, game['home_team_id'])\n",
    "    \n",
    "    df_features = pd.concat([function(dfs_gamestates) for function in functions_features], axis=1)\n",
    "    df_features.to_hdf('features.h5', key=f'game_{game_id}')"
   ]
  },
  {
   "source": [
    "## Generate game state labels"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell lists a number of *label generators* from the `labels` module in the `socceraction` library. Each function expects either a `DataFrame` object containing actions (i.e., individual actions) or a list of `DataFrame` objects containing consecutive actions (i.e., game states), and returns the corresponding *label* for the individual action or game state."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "functions_labels = [\n",
    "    labels.scores,\n",
    "    labels.concedes\n",
    "]"
   ]
  },
  {
   "source": [
    "The following cell computes the labels for each action:\n",
    "\n",
    "1. Obtain the actions for the game (i.e., `df_actions`) by looping through the games;\n",
    "2. Compute the labels for the actions (i.e., `df_labels`) by looping through the list of *label generators*."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "a3a3cfc1841b40dc99a658dccede81b5"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_actions = pd.read_hdf('spadl.h5', key=f'actions/game_{game_id}')\n",
    "    df_actions = (df_actions\n",
    "        .merge(df_actiontypes, how='left')\n",
    "        .merge(df_results, how='left')\n",
    "        .merge(df_bodyparts, how='left')\n",
    "        .reset_index(drop=True)\n",
    "    )\n",
    "    \n",
    "    df_labels = pd.concat([function(df_actions) for function in functions_labels], axis=1)\n",
    "    df_labels.to_hdf('labels.h5', key=f'game_{game_id}')"
   ]
  },
  {
   "source": [
    "## Generate dataset"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell generates a list of names for the features to be included in the dataset."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_features = features.feature_column_names(functions_features, nb_prev_actions=nb_prev_actions)"
   ]
  },
  {
   "source": [
    "The following cell obtains the relevant features for each game and stores them in the `df_features` `DataFrame` object."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "5255b25660764f56840ee8b5d528101a"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "dfs_features = []\n",
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_features = pd.read_hdf('features.h5', key=f'game_{game_id}')\n",
    "    dfs_features.append(df_features[columns_features])\n",
    "df_features = pd.concat(dfs_features).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   type_pass_a0  type_cross_a0  type_throw_in_a0  type_freekick_crossed_a0  \\\n0          True          False             False                     False   \n1          True          False             False                     False   \n2          True          False             False                     False   \n3          True          False             False                     False   \n4          True          False             False                     False   \n5          True          False             False                     False   \n6         False           True             False                     False   \n7         False          False             False                     False   \n8         False           True             False                     False   \n9         False          False             False                     False   \n\n   type_freekick_short_a0  type_corner_crossed_a0  type_corner_short_a0  \\\n0                   False                   False                 False   \n1                   False                   False                 False   \n2                   False                   False                 False   \n3                   False                   False                 False   \n4                   False                   False                 False   \n5                   False                   False                 False   \n6                   False                   False                 False   \n7                   False                   False                 False   \n8                   False                   False                 False   \n9                   False                   False                 False   \n\n   type_take_on_a0  type_foul_a0  type_tackle_a0  ...  end_dist_to_goal_a0  \\\n0            False         False           False  ...            63.091679   \n1            False         False           False  ...            68.328929   \n2            False         False           False  ...            73.715416   \n3            False         False           False  ...            38.707772   \n4            False         False           False  ...            37.425928   \n5            False         False           False  ...            30.729141   \n6            False         False           False  ...             8.400000   \n7            False         False           False  ...           106.094842   \n8            False         False           False  ...            25.925192   \n9            False         False           False  ...            89.647658   \n\n   end_angle_to_goal_a0  end_dist_to_goal_a1  end_angle_to_goal_a1  \\\n0              0.053916            63.091679              0.053916   \n1              0.355773            63.091679              0.053916   \n2              0.185556            68.328929              0.355773   \n3              0.396818            73.715416              0.185556   \n4              0.620467            38.707772              0.396818   \n5              1.258205            37.425928              0.620467   \n6              0.000000            30.729141              1.258205   \n7              0.246030            96.600000              0.000000   \n8              1.489705            25.925192              1.489705   \n9              0.388999           106.094842              0.246030   \n\n   end_dist_to_goal_a2  end_angle_to_goal_a2  team_1  team_2  time_delta_1  \\\n0            63.091679              0.053916    True    True      0.000000   \n1            63.091679              0.053916    True    True      1.997756   \n2            63.091679              0.053916    True    True      0.771744   \n3            68.328929              0.355773    True    True      2.174464   \n4            73.715416              0.185556    True    True      3.907382   \n5            38.707772              0.396818    True    True      3.758730   \n6            37.425928              0.620467    True    True      2.210584   \n7            99.923872              0.296969   False   False      1.756122   \n8             8.400000              0.000000   False    True      2.095783   \n9           106.094842              0.246030   False    True      3.034782   \n\n   time_delta_2  \n0      0.000000  \n1      1.997756  \n2      2.769500  \n3      2.946208  \n4      6.081846  \n5      7.666112  \n6      5.969314  \n7      3.966706  \n8      3.851905  \n9      5.130565  \n\n[10 rows x 142 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>type_pass_a0</th>\n      <th>type_cross_a0</th>\n      <th>type_throw_in_a0</th>\n      <th>type_freekick_crossed_a0</th>\n      <th>type_freekick_short_a0</th>\n      <th>type_corner_crossed_a0</th>\n      <th>type_corner_short_a0</th>\n      <th>type_take_on_a0</th>\n      <th>type_foul_a0</th>\n      <th>type_tackle_a0</th>\n      <th>...</th>\n      <th>end_dist_to_goal_a0</th>\n      <th>end_angle_to_goal_a0</th>\n      <th>end_dist_to_goal_a1</th>\n      <th>end_angle_to_goal_a1</th>\n      <th>end_dist_to_goal_a2</th>\n      <th>end_angle_to_goal_a2</th>\n      <th>team_1</th>\n      <th>team_2</th>\n      <th>time_delta_1</th>\n      <th>time_delta_2</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>63.091679</td>\n      <td>0.053916</td>\n      <td>63.091679</td>\n      <td>0.053916</td>\n      <td>63.091679</td>\n      <td>0.053916</td>\n      <td>True</td>\n      <td>True</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>68.328929</td>\n      <td>0.355773</td>\n      <td>63.091679</td>\n      <td>0.053916</td>\n      <td>63.091679</td>\n      <td>0.053916</td>\n      <td>True</td>\n      <td>True</td>\n      <td>1.997756</td>\n      <td>1.997756</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>73.715416</td>\n      <td>0.185556</td>\n      <td>68.328929</td>\n      <td>0.355773</td>\n      <td>63.091679</td>\n      <td>0.053916</td>\n      <td>True</td>\n      <td>True</td>\n      <td>0.771744</td>\n      <td>2.769500</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>38.707772</td>\n      <td>0.396818</td>\n      <td>73.715416</td>\n      <td>0.185556</td>\n      <td>68.328929</td>\n      <td>0.355773</td>\n      <td>True</td>\n      <td>True</td>\n      <td>2.174464</td>\n      <td>2.946208</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>37.425928</td>\n      <td>0.620467</td>\n      <td>38.707772</td>\n      <td>0.396818</td>\n      <td>73.715416</td>\n      <td>0.185556</td>\n      <td>True</td>\n      <td>True</td>\n      <td>3.907382</td>\n      <td>6.081846</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>30.729141</td>\n      <td>1.258205</td>\n      <td>37.425928</td>\n      <td>0.620467</td>\n      <td>38.707772</td>\n      <td>0.396818</td>\n      <td>True</td>\n      <td>True</td>\n      <td>3.758730</td>\n      <td>7.666112</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>False</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>8.400000</td>\n      <td>0.000000</td>\n      <td>30.729141</td>\n      <td>1.258205</td>\n      <td>37.425928</td>\n      <td>0.620467</td>\n      <td>True</td>\n      <td>True</td>\n      <td>2.210584</td>\n      <td>5.969314</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>106.094842</td>\n      <td>0.246030</td>\n      <td>96.600000</td>\n      <td>0.000000</td>\n      <td>99.923872</td>\n      <td>0.296969</td>\n      <td>False</td>\n      <td>False</td>\n      <td>1.756122</td>\n      <td>3.966706</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>False</td>\n      <td>True</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>25.925192</td>\n      <td>1.489705</td>\n      <td>25.925192</td>\n      <td>1.489705</td>\n      <td>8.400000</td>\n      <td>0.000000</td>\n      <td>False</td>\n      <td>True</td>\n      <td>2.095783</td>\n      <td>3.851905</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>False</td>\n      <td>...</td>\n      <td>89.647658</td>\n      <td>0.388999</td>\n      <td>106.094842</td>\n      <td>0.246030</td>\n      <td>106.094842</td>\n      <td>0.246030</td>\n      <td>False</td>\n      <td>True</td>\n      <td>3.034782</td>\n      <td>5.130565</td>\n    </tr>\n  </tbody>\n</table>\n<p>10 rows × 142 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 25
    }
   ],
   "source": [
    "df_features.head(10)"
   ]
  },
  {
   "source": [
    "The following cell lists the names of the labels to be included in the dataset."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_labels = [\n",
    "    'scores',\n",
    "    'concedes'\n",
    "]"
   ]
  },
  {
   "source": [
    "The following cell obtains the relevant labels for each game and stores them in the `df_labels` `DataFrame` object."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "497f936856344641b29144de86c98d96"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "dfs_labels = []\n",
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_labels = pd.read_hdf('labels.h5', key=f'game_{game_id}')\n",
    "    dfs_labels.append(df_labels[columns_labels])\n",
    "df_labels = pd.concat(dfs_labels).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   scores  concedes\n0   False     False\n1   False     False\n2   False     False\n3   False     False\n4   False     False\n5   False     False\n6   False     False\n7   False     False\n8   False     False\n9   False     False",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>scores</th>\n      <th>concedes</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>False</td>\n      <td>False</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 28
    }
   ],
   "source": [
    "df_labels.head(10)"
   ]
  },
  {
   "source": [
    "## Train classifiers"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell trains an XGBoost classifier for each label using the computed features. For each label:\n",
    "1. Construct an XGBoost classifier with default hyperparameters;\n",
    "2. Train the classifier using the computed features and the label;\n",
    "3. Store the trained classifier in the `models` `dict`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "Wall time: 10min 33s\nParser   : 189 ms\n"
    }
   ],
   "source": [
    "%%time\n",
    "models = {}\n",
    "for column_labels in columns_labels:\n",
    "    model = XGBClassifier()\n",
    "    model.fit(df_features, df_labels[column_labels])\n",
    "    models[column_labels] = model"
   ]
  },
  {
   "source": [
    "## Estimate probabilities"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "The following cell predicts the labels for the game states using the trained XGBoost classifier. For each label:\n",
    "1. Retrieve the model for the label<br>\n",
    "2. Estimate the probabilities of the labels being `False` and `True` given the computed features<br>\n",
    "3. Keep the probabilities for the `True` label<br>\n",
    "4. Store the probabilities as a `Series` object in the `dfs_predictions` `dict`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs_predictions = {}\n",
    "for column_labels in columns_labels:\n",
    "    model = models[column_labels]\n",
    "    probabilities = model.predict_proba(df_features)\n",
    "    predictions = probabilities[:, 1]\n",
    "    dfs_predictions[column_labels] = pd.Series(predictions)\n",
    "df_predictions = pd.concat(dfs_predictions, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "     scores  concedes\n0  0.001565  0.000342\n1  0.002807  0.001644\n2  0.002975  0.000755\n3  0.009620  0.001625\n4  0.017388  0.000717\n5  0.010581  0.007906\n6  0.005190  0.001946\n7  0.003626  0.019720\n8  0.002979  0.001047\n9  0.001261  0.011917",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>scores</th>\n      <th>concedes</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0.001565</td>\n      <td>0.000342</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0.002807</td>\n      <td>0.001644</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0.002975</td>\n      <td>0.000755</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.009620</td>\n      <td>0.001625</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.017388</td>\n      <td>0.000717</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>0.010581</td>\n      <td>0.007906</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>0.005190</td>\n      <td>0.001946</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>0.003626</td>\n      <td>0.019720</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>0.002979</td>\n      <td>0.001047</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>0.001261</td>\n      <td>0.011917</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 31
    }
   ],
   "source": [
    "df_predictions.head(10)"
   ]
  },
  {
   "source": [
    "The following cell obtains the `game_id` for each action in order to store the predictions per game."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "f9941be6b02e4ca7b45cfcce9c63dcdd"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "dfs_game_ids = []\n",
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_actions = pd.read_hdf('spadl.h5', key=f'actions/game_{game_id}')\n",
    "    dfs_game_ids.append(df_actions['game_id'])\n",
    "df_game_ids = pd.concat(dfs_game_ids, axis=0).astype('int').reset_index(drop=True)"
   ]
  },
  {
   "source": [
    "The following cell concatenates the `DataFrame` objects with predictions and `game_id`s for each action into a single `DataFrame` object."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predictions = pd.concat([df_predictions, df_game_ids], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "     scores  concedes  game_id\n0  0.001565  0.000342  2500089\n1  0.002807  0.001644  2500089\n2  0.002975  0.000755  2500089\n3  0.009620  0.001625  2500089\n4  0.017388  0.000717  2500089\n5  0.010581  0.007906  2500089\n6  0.005190  0.001946  2500089\n7  0.003626  0.019720  2500089\n8  0.002979  0.001047  2500089\n9  0.001261  0.011917  2500089",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>scores</th>\n      <th>concedes</th>\n      <th>game_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0.001565</td>\n      <td>0.000342</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0.002807</td>\n      <td>0.001644</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0.002975</td>\n      <td>0.000755</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.009620</td>\n      <td>0.001625</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.017388</td>\n      <td>0.000717</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>0.010581</td>\n      <td>0.007906</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>0.005190</td>\n      <td>0.001946</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>0.003626</td>\n      <td>0.019720</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>0.002979</td>\n      <td>0.001047</td>\n      <td>2500089</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>0.001261</td>\n      <td>0.011917</td>\n      <td>2500089</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 34
    }
   ],
   "source": [
    "df_predictions.head(10)"
   ]
  },
  {
   "source": [
    "The following cell groups the predictions per game based on their `game_id`."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_predictions_per_game = df_predictions.groupby('game_id')"
   ]
  },
  {
   "source": [
    "The following cell stores the predictions in the `predictions.h5` HDF5 file per game."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "b4c5d186f13d48cf82e2d7bb6780bafd"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "for game_id, df_predictions in tqdm(df_predictions_per_game):\n",
    "    df_predictions = df_predictions.reset_index(drop=True)\n",
    "    df_predictions[columns_labels].to_hdf('predictions.h5', key=f'game_{game_id}')"
   ]
  },
  {
   "source": [
    "# Value on-the-ball actions"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "**Note:** The code in this section is based on the [4-compute-vaep-values.ipynb](https://github.com/ML-KULeuven/socceraction/blob/master/public-notebooks/4-compute-vaep-values.ipynb) notebook in the `socceraction` repository."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_players = pd.read_hdf('spadl.h5', key='players')\n",
    "df_teams = pd.read_hdf('spadl.h5', key='teams')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "output_type": "display_data",
     "data": {
      "text/plain": "HBox(children=(FloatProgress(value=0.0, max=380.0), HTML(value='')))",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "44eb96c9b1d84fd187a13b684fd71bca"
      }
     },
     "metadata": {}
    },
    {
     "output_type": "stream",
     "name": "stdout",
     "text": "\n"
    }
   ],
   "source": [
    "dfs_values = []\n",
    "for _, game in tqdm(df_games.iterrows(), total=len(df_games)):\n",
    "    game_id = game['game_id']\n",
    "    df_actions = pd.read_hdf('spadl.h5', key=f'actions/game_{game_id}')\n",
    "    df_actions = (df_actions\n",
    "        .merge(df_actiontypes, how='left')\n",
    "        .merge(df_results, how='left')\n",
    "        .merge(df_bodyparts, how='left')\n",
    "        .merge(df_players, how='left')\n",
    "        .merge(df_teams, how='left')\n",
    "        .reset_index(drop=True)\n",
    "    )\n",
    "    \n",
    "    df_predictions = pd.read_hdf('predictions.h5', key=f'game_{game_id}')\n",
    "    df_values = value(df_actions, df_predictions['scores'], df_predictions['concedes'])\n",
    "    \n",
    "    df_all = pd.concat([df_actions, df_predictions, df_values], axis=1)\n",
    "    dfs_values.append(df_all)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_values = (pd.concat(dfs_values)\n",
    "    .sort_values(['game_id', 'period_id', 'time_seconds'])\n",
    "    .reset_index(drop=True)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "       short_name    scores  concedes  offensive_value  defensive_value  \\\n0    A. Lacazette  0.000207  0.000070         0.000000        -0.000000   \n1      R. Holding  0.005683  0.000795         0.005476        -0.000725   \n2         M. Özil  0.007332  0.000862         0.001649        -0.000068   \n3  Mohamed Elneny  0.004931  0.001380        -0.002401        -0.000518   \n4        Bellerín  0.009885  0.003907         0.004954        -0.002527   \n5         M. Özil  0.008336  0.003977        -0.001549        -0.000070   \n6      H. Maguire  0.004430  0.006996         0.000453         0.001340   \n7        Bellerín  0.008783  0.003025         0.001787         0.001405   \n8        Bellerín  0.008157  0.002096        -0.000626         0.000929   \n9        G. Xhaka  0.006883  0.001196        -0.001274         0.000900   \n\n   vaep_value  \n0    0.000000  \n1    0.004751  \n2    0.001582  \n3   -0.002919  \n4    0.002427  \n5   -0.001619  \n6    0.001793  \n7    0.003192  \n8    0.000303  \n9   -0.000374  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>short_name</th>\n      <th>scores</th>\n      <th>concedes</th>\n      <th>offensive_value</th>\n      <th>defensive_value</th>\n      <th>vaep_value</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A. Lacazette</td>\n      <td>0.000207</td>\n      <td>0.000070</td>\n      <td>0.000000</td>\n      <td>-0.000000</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>R. Holding</td>\n      <td>0.005683</td>\n      <td>0.000795</td>\n      <td>0.005476</td>\n      <td>-0.000725</td>\n      <td>0.004751</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>M. Özil</td>\n      <td>0.007332</td>\n      <td>0.000862</td>\n      <td>0.001649</td>\n      <td>-0.000068</td>\n      <td>0.001582</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Mohamed Elneny</td>\n      <td>0.004931</td>\n      <td>0.001380</td>\n      <td>-0.002401</td>\n      <td>-0.000518</td>\n      <td>-0.002919</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Bellerín</td>\n      <td>0.009885</td>\n      <td>0.003907</td>\n      <td>0.004954</td>\n      <td>-0.002527</td>\n      <td>0.002427</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>M. Özil</td>\n      <td>0.008336</td>\n      <td>0.003977</td>\n      <td>-0.001549</td>\n      <td>-0.000070</td>\n      <td>-0.001619</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>H. Maguire</td>\n      <td>0.004430</td>\n      <td>0.006996</td>\n      <td>0.000453</td>\n      <td>0.001340</td>\n      <td>0.001793</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Bellerín</td>\n      <td>0.008783</td>\n      <td>0.003025</td>\n      <td>0.001787</td>\n      <td>0.001405</td>\n      <td>0.003192</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Bellerín</td>\n      <td>0.008157</td>\n      <td>0.002096</td>\n      <td>-0.000626</td>\n      <td>0.000929</td>\n      <td>0.000303</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>G. Xhaka</td>\n      <td>0.006883</td>\n      <td>0.001196</td>\n      <td>-0.001274</td>\n      <td>0.000900</td>\n      <td>-0.000374</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 40
    }
   ],
   "source": [
    "df_values[\n",
    "    ['short_name', 'scores', 'concedes', 'offensive_value', 'defensive_value', 'vaep_value']\n",
    "].head(10)"
   ]
  },
  {
   "source": [
    "# Rate players"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "**Note:** The code in this section is based on the [5-top-players.ipynb](https://github.com/ML-KULeuven/socceraction/blob/master/public-notebooks/5-top-players.ipynb) notebook in the `socceraction` repository."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Rate according to total VAEP value"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ranking = (df_values[['player_id', 'team_name', 'short_name', 'vaep_value']]\n",
    "    .groupby(['player_id', 'team_name', 'short_name'])\n",
    "    .agg(vaep_count=('vaep_value', 'count'), vaep_sum=('vaep_value', 'sum'))\n",
    "    .sort_values('vaep_sum', ascending=False)\n",
    "    .reset_index()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   player_id             team_name       short_name  vaep_count   vaep_sum\n0   120353.0          Liverpool FC    Mohamed Salah        1568  23.858101\n1    38021.0    Manchester City FC     K. De Bruyne        3528  19.006954\n2    26150.0     Leicester City FC        R. Mahrez        2022  18.335896\n3       54.0  Tottenham Hotspur FC       C. Eriksen        2959  16.920042\n4    25707.0            Chelsea FC        E. Hazard        1974  16.024981\n5     8717.0  Tottenham Hotspur FC          H. Kane        1153  14.096480\n6     8317.0    Manchester City FC      David Silva        2925  12.788353\n7    15808.0          Liverpool FC  Roberto Firmino        1633  11.886950\n8    49872.0         Stoke City FC       X. Shaqiri        1722  11.708034\n9     3319.0            Arsenal FC          M. Özil        2278  11.278624",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>player_id</th>\n      <th>team_name</th>\n      <th>short_name</th>\n      <th>vaep_count</th>\n      <th>vaep_sum</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>120353.0</td>\n      <td>Liverpool FC</td>\n      <td>Mohamed Salah</td>\n      <td>1568</td>\n      <td>23.858101</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>38021.0</td>\n      <td>Manchester City FC</td>\n      <td>K. De Bruyne</td>\n      <td>3528</td>\n      <td>19.006954</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>26150.0</td>\n      <td>Leicester City FC</td>\n      <td>R. Mahrez</td>\n      <td>2022</td>\n      <td>18.335896</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>54.0</td>\n      <td>Tottenham Hotspur FC</td>\n      <td>C. Eriksen</td>\n      <td>2959</td>\n      <td>16.920042</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>25707.0</td>\n      <td>Chelsea FC</td>\n      <td>E. Hazard</td>\n      <td>1974</td>\n      <td>16.024981</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>8717.0</td>\n      <td>Tottenham Hotspur FC</td>\n      <td>H. Kane</td>\n      <td>1153</td>\n      <td>14.096480</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>8317.0</td>\n      <td>Manchester City FC</td>\n      <td>David Silva</td>\n      <td>2925</td>\n      <td>12.788353</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>15808.0</td>\n      <td>Liverpool FC</td>\n      <td>Roberto Firmino</td>\n      <td>1633</td>\n      <td>11.886950</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>49872.0</td>\n      <td>Stoke City FC</td>\n      <td>X. Shaqiri</td>\n      <td>1722</td>\n      <td>11.708034</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>3319.0</td>\n      <td>Arsenal FC</td>\n      <td>M. Özil</td>\n      <td>2278</td>\n      <td>11.278624</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 42
    }
   ],
   "source": [
    "df_ranking.head(10)"
   ]
  },
  {
   "source": [
    "## Rate according to total VAEP value per 90 minutes"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_player_games = pd.read_hdf('spadl.h5', 'player_games')\n",
    "df_player_games = df_player_games[df_player_games['game_id'].isin(df_games['game_id'])]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_minutes_played = (df_player_games[['player_id', 'minutes_played']]\n",
    "    .groupby('player_id')\n",
    "    .sum()\n",
    "    .reset_index()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "   player_id  minutes_played\n0         36     1238.400194\n1         38      382.741398\n2         48     3344.850202\n3         54     3350.424005\n4         56      267.266792\n5         74     1515.130449\n6         93     2941.903193\n7        107     2839.106630\n8        116     2497.512961\n9        120      234.603990",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>player_id</th>\n      <th>minutes_played</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>36</td>\n      <td>1238.400194</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>38</td>\n      <td>382.741398</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>48</td>\n      <td>3344.850202</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>54</td>\n      <td>3350.424005</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>56</td>\n      <td>267.266792</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>74</td>\n      <td>1515.130449</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>93</td>\n      <td>2941.903193</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>107</td>\n      <td>2839.106630</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>116</td>\n      <td>2497.512961</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>120</td>\n      <td>234.603990</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 45
    }
   ],
   "source": [
    "df_minutes_played.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ranking_p90 = df_ranking.merge(df_minutes_played)\n",
    "df_ranking_p90 = df_ranking_p90[df_ranking_p90['minutes_played'] > 360]\n",
    "df_ranking_p90['vaep_rating'] = df_ranking_p90['vaep_sum'] * 90 / df_ranking_p90['minutes_played']\n",
    "df_ranking_p90 = df_ranking_p90.sort_values('vaep_rating', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": "     player_id             team_name         short_name  vaep_count  \\\n24      3802.0          Liverpool FC  Philippe Coutinho        1014   \n0     120353.0          Liverpool FC      Mohamed Salah        1568   \n4      25707.0            Chelsea FC          E. Hazard        1974   \n2      26150.0     Leicester City FC          R. Mahrez        2022   \n1      38021.0    Manchester City FC       K. De Bruyne        3528   \n151     8249.0  Manchester United FC        M. Fellaini         477   \n209    25865.0     Crystal Palace FC            B. Sako         235   \n6       8317.0    Manchester City FC        David Silva        2925   \n3         54.0  Tottenham Hotspur FC         C. Eriksen        2959   \n9       3319.0            Arsenal FC            M. Özil        2278   \n\n      vaep_sum  minutes_played  vaep_rating  \n24    9.460539     1134.538547     0.750480  \n0    23.858101     2996.529579     0.716572  \n4    16.024981     2505.438518     0.575647  \n2    18.335896     3063.065477     0.538751  \n1    19.006954     3190.369684     0.536184  \n151   3.763032      693.521021     0.488338  \n209   2.884128      535.489797     0.484737  \n6    12.788353     2519.126467     0.456885  \n3    16.920042     3350.424005     0.454511  \n9    11.278624     2253.492156     0.450446  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>player_id</th>\n      <th>team_name</th>\n      <th>short_name</th>\n      <th>vaep_count</th>\n      <th>vaep_sum</th>\n      <th>minutes_played</th>\n      <th>vaep_rating</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>24</th>\n      <td>3802.0</td>\n      <td>Liverpool FC</td>\n      <td>Philippe Coutinho</td>\n      <td>1014</td>\n      <td>9.460539</td>\n      <td>1134.538547</td>\n      <td>0.750480</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>120353.0</td>\n      <td>Liverpool FC</td>\n      <td>Mohamed Salah</td>\n      <td>1568</td>\n      <td>23.858101</td>\n      <td>2996.529579</td>\n      <td>0.716572</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>25707.0</td>\n      <td>Chelsea FC</td>\n      <td>E. Hazard</td>\n      <td>1974</td>\n      <td>16.024981</td>\n      <td>2505.438518</td>\n      <td>0.575647</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>26150.0</td>\n      <td>Leicester City FC</td>\n      <td>R. Mahrez</td>\n      <td>2022</td>\n      <td>18.335896</td>\n      <td>3063.065477</td>\n      <td>0.538751</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>38021.0</td>\n      <td>Manchester City FC</td>\n      <td>K. De Bruyne</td>\n      <td>3528</td>\n      <td>19.006954</td>\n      <td>3190.369684</td>\n      <td>0.536184</td>\n    </tr>\n    <tr>\n      <th>151</th>\n      <td>8249.0</td>\n      <td>Manchester United FC</td>\n      <td>M. Fellaini</td>\n      <td>477</td>\n      <td>3.763032</td>\n      <td>693.521021</td>\n      <td>0.488338</td>\n    </tr>\n    <tr>\n      <th>209</th>\n      <td>25865.0</td>\n      <td>Crystal Palace FC</td>\n      <td>B. Sako</td>\n      <td>235</td>\n      <td>2.884128</td>\n      <td>535.489797</td>\n      <td>0.484737</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>8317.0</td>\n      <td>Manchester City FC</td>\n      <td>David Silva</td>\n      <td>2925</td>\n      <td>12.788353</td>\n      <td>2519.126467</td>\n      <td>0.456885</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>54.0</td>\n      <td>Tottenham Hotspur FC</td>\n      <td>C. Eriksen</td>\n      <td>2959</td>\n      <td>16.920042</td>\n      <td>3350.424005</td>\n      <td>0.454511</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>3319.0</td>\n      <td>Arsenal FC</td>\n      <td>M. Özil</td>\n      <td>2278</td>\n      <td>11.278624</td>\n      <td>2253.492156</td>\n      <td>0.450446</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 47
    }
   ],
   "source": [
    "df_ranking_p90.head(10)"
   ]
  },
  {
   "source": [
    "**References:**\n",
    "* Tom Decroos, Lotte Bransen, Jan Van Haaren, and Jesse Davis. \"[Actions Speak Louder than Goals: Valuing Player Actions in Soccer.](https://arxiv.org/abs/1802.07127)\" In *Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining*, pp. 1851-1861. 2019.\n",
    "* Luca Pappalardo, Paolo Cintia, Alessio Rossi, Emanuele Massucco, Paolo Ferragina, Dino Pedreschi, and Fosca Giannotti. \"[A Public Data Set of Spatio-Temporal Match Events in Soccer Competitions.](https://www.nature.com/articles/s41597-019-0247-7)\" *Scientific Data 6*, no. 1 (2019): 1-15."
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ]
}