webscraping_advanced.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/JosephBless/awesome-colab-notebooks/blob/main/webscraping_advanced.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5IYkERHw-Q97"
      },
      "source": [
        "# Web Scraping Advanced (oDCM)\n",
        "\n",
        "*We finally move towards some real-world examples: Twitter & Instagram! You'll quickly realize that both websites face some extra challenges in terms of scraping. We'll show you how to overcome them, and build a scraper that you can use for your own project!*\n",
        "\n",
        "--- \n",
        "\n",
        "## Learning Objectives\n",
        "\n",
        "Students will be able to: \n",
        "* Understand the difference between headless and browser emulation and ability to apply both methods (using Selenium)\n",
        "* Emulate user interaction with a site using timers, clicks, scrolling, and filling in forms \n",
        "* Access data that is hidden behind a login-screen\n",
        "* Apply search parameters to obtain subsets of data\n",
        "* Scrape and store images from the internet locally\n",
        " \n",
        "\n",
        "--- \n",
        "\n",
        "## Acknowledgements\n",
        "This course draws on a variety of online resources that can be retrieved from the [course website](https://odcm.hannesdatta.com/docs/about/).\n",
        "\n",
        "--- \n",
        "\n",
        "## Contact\n",
        "For technical issues outside of scheduled classes, please check the [support section](https://odcm.hannesdatta.com/docs/course/support) on the course website."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0F81lcj8-Q-B"
      },
      "source": [
        "---\n",
        "## 1. Selenium \n",
        "\n",
        "### 1.1 Why Selenium? \n",
        "In the Web Scraping 101 tutorial, we used BeautifulSoup to turn HTML into a data structure that we could search and access using Python-like syntax. While it's easy to get started with this library, it has limitations when it comes to dynamic websites. That is, websites of which the content changes after each page refresh. Selenium can handle both static and dynamic websites and mimic user behavior (e.g., scrolling, clicking, logging in). It launches another web browser window in which all actions are visible which makes it feel more intuitive. For example, the video below launches a regular Google Chrome window and visits [`instagram.com`](https://www.instagram.com). This browser window behaves like normal, so you can click on buttons and fill out fields. Yet you can distinguish it from your normal web browser by the header that indicates that Chrome is being controlled by automated test software. Before you can try it out yourself, we need to install some additional software which we'll explain next. \n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/hannesdatta/course-odcm/dev/content/docs/tutorials/webscrapingadvanced/images/selenium_instagram.gif\" align=\"left\" width=70%/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O-5XmefK-Q-C"
      },
      "source": [
        "---\n",
        "### 1.2 Installing Selenium\n",
        "\n",
        "#### Using Selenium with Jupyter Notebook on your own computer\n",
        "You will need to (1) install the Python package for Selenium, (2) download a web driver to interface with a web browser, and (3) configure Selenium to recognize your web driver.\n",
        "1. Open Anaconda Prompt (Windows) or the Terminal (Mac), type the command `conda install selenium`, and agree to whatever the package manager wants to install or update (usually by pressing `y` to confirm your choice). \n",
        "2. Once we run the scraper, a Chrome browser launches which requires a web driver executable file. Download this file from [here](https://sites.google.com/a/chromium.org/chromedriver/downloads) (open [this](https://www.whatismybrowser.com/detect/what-version-of-chrome-do-i-have) site in Chrome to identify your current Chrome version). \n",
        "3. Unzip the file and move it to the same directory where you're running this notebook. Either you can set a `chrome_path` manually (e.g., `/Users/royklaassebos/Google Drive/2020/Data_Scientist_Tilburg/oDCM/chromedriver` in my case) or ideally you make Chromedriver available through a so-called PATH variable so that you can access it regardless of your current directory. To do so, ...\n",
        "    * Windows users can follow the steps described [here](https://tilburgsciencehub.com/configure/python-for-scraping). If that doesn't work for you, [this](https://zwbetz.com/download-chromedriver-binary-and-add-to-your-path-for-automated-functional-testing/) guide may help.\n",
        "    * Mac users need to move the `chromedriver` file to the `/usr/local/bin` folder. `cd` to the directory of the `chromedriver` file and type `mv chromedriver /usr/local/bin`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ooZOOYyk-Q-E"
      },
      "outputs": [],
      "source": [
        "# only run if you have installed Selenium on your local computer!\n",
        "import selenium.webdriver\n",
        "\n",
        "driver = selenium.webdriver.Chrome()\n",
        "    \n",
        "# if above solutions did not work or you ran into problems with the PATH variable, try the following: driver = selenium.webdriver.Chrome(chrome_path = <PATH_TO_FILE>)\n",
        "# tip: if your Chromedriver chrashes or does not work as expected throughout the exercises, run this cell again to create a new instance of the chromedriver"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5m-cL5L-Q-F"
      },
      "source": [
        "#### Using Selenium with Google Colab"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hnnw3jtN-Q-G"
      },
      "outputs": [],
      "source": [
        "# Running this notebook on Google Colab? Then run this cell! Otherwise, please use the cell above!\n",
        "\n",
        "!pip install selenium\n",
        "!apt-get update \n",
        "!apt install chromium-chromedriver\n",
        "\n",
        "from selenium import webdriver\n",
        "chrome_options = webdriver.ChromeOptions()\n",
        "chrome_options.add_argument('--headless')\n",
        "chrome_options.add_argument('--no-sandbox')\n",
        "chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ET9H7nTx-Q-G"
      },
      "source": [
        "**Let's try it out!**  \n",
        "Follow the steps and make sure it works properly on your machine. What happens once you run the cell above twice? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9w3I36Zt-Q-H"
      },
      "source": [
        "---\n",
        "### 1.3 Access Sites Programmatically"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OWgoXTG4-Q-H"
      },
      "source": [
        "**Importance**  \n",
        "Next, we're going to tell the browser to visit the Tilburg University Twitter account. We call the `driver` object we created above and use the `get` method, which we pass the URL of the website we'd like to extract. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "awCV2U0E-Q-I"
      },
      "outputs": [],
      "source": [
        "driver.get(\"https://twitter.com/TilburgU\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rAd9ihxX-Q-I"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/hannesdatta/course-odcm/dev/content/docs/tutorials/webscrapingadvanced/images/twitter_tilburgu.png\" align=\"left\" width=40%/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ml-KKmyS-Q-I"
      },
      "source": [
        "**Let's try it out!**  \n",
        "As most information can only be obtained once you're signed in, manually login to your Twitter account through the driver page (create a new account if you don't have one yet - or if you don't want to run the risk of getting blocked on your personal account). \n",
        "\n",
        "From this point, we can use BeautifulSoup as we learned previously, though we create the `res` object from the `driver` object this time. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8eSjGAw6-Q-I"
      },
      "outputs": [],
      "source": [
        "# make sure to login to your Twitter account first!\n",
        "from bs4 import BeautifulSoup\n",
        "res = driver.page_source.encode('utf-8')\n",
        "soup = BeautifulSoup(res, \"html.parser\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MHHIXjHm-Q-J"
      },
      "source": [
        "Once you inspect the HTML code of the Twitter page you'll discover that the class names are more complex than the ones we looked at earlier. Take a look at the gigantic class name of the Twitter bio, for example..."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKOvclqA-Q-J"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/hannesdatta/course-odcm/dev/content/docs/tutorials/webscrapingadvanced/images/twitter_bio.png\" align=\"left\" width=60%/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5a7Iu36Y-Q-J"
      },
      "source": [
        "... and more importantly: these class names are dynamic. If you open up the Inspector and look at the class name right now, the class name is likely going to be different than the one above (check for yourself!). Therefore, you may want to look out for elements that are not subjective to changes such as the `data-testid` attribute."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SX3Eh7_D-Q-K"
      },
      "outputs": [],
      "source": [
        "soup.find(attrs={\"data-testid\": \"UserDescription\"}).text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mf72xf4y-Q-K"
      },
      "source": [
        "**Exercise 1**  \n",
        "Using the same approach as above, extract the (i) number of followers, (ii) the location, and the (iii) join date of the [TilburgU](https://twitter.com/tilburgU) Twitter account. Tip: use Google Inspector to determine an appropriate navigation strategy. Take into consideration the dynamic nature of class names; so look for ways to navigate the source code without relying on these temporary classes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SN5CXVZs-Q-K"
      },
      "outputs": [],
      "source": [
        "# your answer goes here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9e8-t6vc-Q-K"
      },
      "outputs": [],
      "source": [
        "# solution\n",
        "followers = soup.find(attrs = {\"href\": \"/TilburgU/followers\"}).find_all('span')[1].text\n",
        "location = soup.find(attrs={\"data-testid\": \"UserProfileHeader_Items\"}).find_all('span')[1].text\n",
        "join_date = soup.find(attrs = {\"data-testid\": \"UserProfileHeader_Items\"}).find_all('span')[3].text\n",
        "\n",
        "print(f\"Followers: {followers} \\nLocation: {location} \\nJoin date: {join_date}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oEEn7ELe-Q-L"
      },
      "source": [
        "---\n",
        "### 1.4 Scroll Sites Programmatically\n",
        "\n",
        "**Importance**  \n",
        "In a similar way, we can scrape the content of the most recent tweet as follows: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mOL7_Yiu-Q-L"
      },
      "outputs": [],
      "source": [
        "# 1st tweet\n",
        "soup.find_all(attrs={\"data-testid\": \"tweet\"})[0].find_all(attrs={\"dir\": \"auto\"})[4].text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IfbxX60V-Q-L"
      },
      "source": [
        "And for older tweets we simply increment the counter by one: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5Mk1lipL-Q-L"
      },
      "outputs": [],
      "source": [
        "# 2nd tweet\n",
        "soup.find_all(attrs={\"data-testid\": \"tweet\"})[1].find_all(attrs={\"dir\": \"auto\"})[4].text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z00r2zcy-Q-L"
      },
      "source": [
        "Easy right? Not so fast.. From the 10th tweet onwards (in your case it may be a different figure; dependent on screen size, resolution, etc.), it returns an `IndexError: list index out of range`. This is because Twitter only pulls in new tweets once you scroll down the page. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8fJa1IYF-Q-M"
      },
      "outputs": [],
      "source": [
        "# 10th tweet\n",
        "soup.find_all(attrs={\"data-testid\": \"tweet\"})[9].find_all(attrs={\"dir\": \"auto\"})[4].text"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOCnPtWi-Q-M"
      },
      "source": [
        "Therefore, we need to scroll down to the bottom of the page if we like to obtain more than a few tweets. Every time you run the cell below it loads another 5-10 tweets.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wnb8XRJ_-Q-M"
      },
      "outputs": [],
      "source": [
        "driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U3J1Mbdz-Q-M"
      },
      "source": [
        "**Let's try it out!**  \n",
        "Try running the cell above a couple of times. What happens to the most recent tweets? If you run the cells again that extract the 1st, 2nd, 9th, and 10th tweet, does the output change? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hCkirwPX-Q-M"
      },
      "source": [
        "Indeed, we need to recreate the `res` object after each iteration because the HTML code changes once you scroll down (older tweets are added and newer ones are hidden). The number of tweets in the view deviates depending on the type of media (e.g., images take up more space than text). Therefore, we first determine the number of views in the current view to make sure we capture all tweets. After we stored the last tweet in the view, we scroll down the page and start all over again.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wnKdyh9--Q-M"
      },
      "outputs": [],
      "source": [
        "from time import sleep\n",
        "tweets = []\n",
        "\n",
        "for _ in range(5):\n",
        "    res = driver.page_source.encode('utf-8')\n",
        "    soup = BeautifulSoup(res, \"html.parser\")\n",
        "    \n",
        "    # total number of tweets in current view\n",
        "    num_tweets_view = len(soup.find_all(attrs={\"data-testid\": \"tweet\"}))\n",
        "        \n",
        "    # add tweets to list\n",
        "    for counter in range(num_tweets_view):\n",
        "        tweets.append(soup.find_all(attrs={\"data-testid\": \"tweet\"})[counter].find_all(attrs={\"dir\": \"auto\"})[4].text)\n",
        "    \n",
        "    # scroll down the page\n",
        "    driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')\n",
        "    \n",
        "    # pause for 5 seconds\n",
        "    sleep(1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zg9igg3N-Q-N"
      },
      "source": [
        "**Exercise 2**\n",
        "1. What happens once you first scroll down the page and then run the cell above? Does `tweets` differ? Why? \n",
        "2. Estimate how many times you would need to scroll in order to capture all tweets (tip: you find the total number of tweets at the top). By the way, there's no need to collect all tweets!\n",
        "3. Write a function `process_tweets()` that takes a list of `tweets` as input and returns a CSV file that contains the original tweet, a list of mentions (e.g., `@gemeentetilburg`), and a list of hashtags (e.g., `#makeitintilburg`). We call this \"parsing\"(i.e., the conversion of data from JSON into a CSV file). Tip: you may first want to split each tweet into a list of words and work from there. When is a word considered a hashtag? And a mention? How about punctuation? Test your function with the list of `tweets` above. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z50SK0FA-Q-N"
      },
      "outputs": [],
      "source": [
        "# your answer goes here!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNuBIFTq-Q-N"
      },
      "source": [
        "**Solution**  \n",
        "1. The scraper will start from the current view. Since more recent tweets are hidden as you scroll down, the scraper would skip the first few tweets in that case. \n",
        "2. Scrolling down five times yielded 56 tweets in my case, so it would take about 3986/56 = 71 times on average. Again, the answer may differ for your machine."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X-yjwPo7-Q-N"
      },
      "outputs": [],
      "source": [
        "# Question 3\n",
        "import pandas as pd\n",
        "\n",
        "def process_tweets(tweets):\n",
        "    output = []\n",
        "    \n",
        "    for tweet in tweets: \n",
        "        mentions = []\n",
        "        hashtags = []\n",
        "        \n",
        "        # a more elegant solution can be achieved using regular expressions (outside the scope of this course)\n",
        "        \n",
        "        # remove punctuation (to avoid #hashtag?, #hashtag!, etc.)\n",
        "        for character in [\"?\", \".\", \",\", \"!\"]:\n",
        "            tweet_clean = tweet.replace(character, \"\")\n",
        "            tweet = tweet_clean\n",
        "            \n",
        "        # separate words chained by an enter with a space (to avoid #hashtag\\nABCDEF)\n",
        "        tweet = tweet.replace(\"\\n\", \" \")\n",
        "        \n",
        "        for word in tweet.split(\" \"): \n",
        "            try: \n",
        "                if word[0] == \"@\" and word.count(\"@\") == 1: \n",
        "                    mentions.append(word)\n",
        "                if word[0] == \"#\" and word.count(\"#\") == 1: \n",
        "                    hashtags.append(word) \n",
        "            except: \n",
        "                pass\n",
        "            \n",
        "        output.append({\n",
        "            \"tweet\": tweet,\n",
        "            \"mentions\": mentions,\n",
        "            \"hashtags\": hashtags\n",
        "        })\n",
        "        \n",
        "    df = pd.DataFrame(output)\n",
        "    df.to_csv(\"tweets_mentions_hashtags.csv\", index=False)\n",
        "\n",
        "process_tweets(tweets)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QSyyCgQd-Q-O"
      },
      "source": [
        "**Exercise 3**  \n",
        "Retrieve the list of all accounts Tilburg University is following on Twitter. See: https://twitter.com/TilburgU/following. __Make sure to log in to your Twitter account in the browser that you're using for scraping!__.\n",
        "\n",
        "1. Store their full name, Twitter handle, and biography in a csv-file (`following.csv`). \n",
        "2. Load `following.csv` and determine the percentage of accounts that use the word `Professor` or `professor` in their bio."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMf45wqR-Q-O"
      },
      "source": [
        "**Solutions**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MsK5hM9i-Q-O"
      },
      "outputs": [],
      "source": [
        "# Question 1\n",
        "from time import sleep\n",
        "\n",
        "def twitter_followings():\n",
        "    users = []\n",
        "\n",
        "    # we set the range to 3 here so the code works reasonably fast.\n",
        "    # in practice, you ideally would like to set this to 10, because \n",
        "    # each view contains 10-15 accounts and we know that there are approximately 100 accounts in total\n",
        "    \n",
        "    for _ in range(3): \n",
        "        res = driver.page_source.encode('utf-8')\n",
        "        soup = BeautifulSoup(res, \"html.parser\")\n",
        "        \n",
        "        # if you don't specify the primary column it will also scrape the accounts below \"Who to follow\" (right sidebar)\n",
        "        data = soup.find(attrs={\"data-testid\": \"primaryColumn\"}).find_all(attrs={\"data-testid\": \"UserCell\"})\n",
        "\n",
        "        for counter in range(len(data)):\n",
        "            user = data[counter].find_all(\"span\")\n",
        "\n",
        "            full_name = user[1].text\n",
        "            handle = user[2].text\n",
        "\n",
        "            # not all users have a bio \n",
        "            try: \n",
        "                bio = user[5].text\n",
        "            except: \n",
        "                bio = None\n",
        "\n",
        "            user_data = {\"full_name\": full_name, \n",
        "                          \"handle\": handle,\n",
        "                          \"bio\": bio\n",
        "                         }\n",
        "                \n",
        "            if user_data not in users:  # to filter out potential duplicates\n",
        "                users.append(user_data)\n",
        "                \n",
        "        driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')\n",
        "        sleep(5)\n",
        "         \n",
        "    df = pd.DataFrame(users)\n",
        "    df.to_csv(\"followings.csv\", index=False)\n",
        "              \n",
        "# don't forget to manually login to your Twitter account to access the following list before proceeding\n",
        "driver.get(\"https://twitter.com/TilburgU/following\") \n",
        "\n",
        "# add a short pause here, or Twitter recognizes that it's a scraper and will block your request!\n",
        "sleep(5)\n",
        "\n",
        "twitter_followings()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DRICUlIb-Q-P"
      },
      "outputs": [],
      "source": [
        "# Question 2\n",
        "followings = pd.read_csv(\"followings.csv\")\n",
        "\n",
        "professor_count = 0 \n",
        "\n",
        "for row in followings[\"bio\"]: \n",
        "    try:  # some users don't have a bio which causes an error\n",
        "        if \"professor\" in row.lower():\n",
        "            professor_count += 1\n",
        "    except: \n",
        "        pass\n",
        "\n",
        "print(f\"The percentage of professors among the list of account Tilburg University follows is {round(professor_count / len(followings) * 100, 1)}%.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IKeCEGOd-Q-P"
      },
      "source": [
        "Up to now, we have either picked an arbitrary value for the number of scrolls, or - at best - we have approximated the number of times based on the total number of records. An alternative strategy is based on the idea that the current position on the page remains the same if you're already at the bottom of the page and still try to scroll down. Simply put, if scrolling down changes the current position then we're not at the bottom of the page yet. \n",
        "\n",
        "With this idea in mind, we can implement this procedure with a `while` loop that remains true as long as we have not reached the end. Once the `current_height` equals the height before scrolling down (`last_height`), we `break` out of the loop and print the total number of scrolls (`scroll_counter`):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g1wTnDPd-Q-P"
      },
      "outputs": [],
      "source": [
        "scroll_counter = 0 \n",
        "last_height = 0\n",
        "driver.get(\"https://twitter.com/TilburgU\")\n",
        "\n",
        "# running this cell may take a minute or two\n",
        "while True:    \n",
        "    current_height = driver.execute_script('return document.body.scrollHeight')\n",
        "    print(\"current height: \" + str(current_height))\n",
        "\n",
        "    if current_height == last_height:\n",
        "        break\n",
        "\n",
        "    driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
        "    scroll_counter += 1\n",
        "    last_height = current_height\n",
        "    print(\"last height: \" + str(last_height))\n",
        "    sleep(1)\n",
        "    \n",
        "print(f\"The number scrolls required to scrape all tweets is: {scroll_counter}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "srUtCN4X-Q-P"
      },
      "source": [
        "**Exercise 4**  \n",
        "Extend your answer of Exercise 3 so that the number of scroll is dynamic. That is, it scrolls the minimum number of times required to capture all followings (not tweets!) and updates its value once new accounts are followed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AuO-1frz-Q-Q"
      },
      "outputs": [],
      "source": [
        "# your answer goes here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "scrolled": true,
        "id": "QPzEBHwe-Q-Q"
      },
      "outputs": [],
      "source": [
        "# solution\n",
        "from time import sleep\n",
        "\n",
        "def twitter_followings_while():\n",
        "    users = []\n",
        "    current_height = 0\n",
        "    last_height = 0\n",
        "    \n",
        "    while True: \n",
        "        current_height = driver.execute_script('return document.body.scrollHeight')\n",
        "              \n",
        "        if current_height == last_height: \n",
        "            break\n",
        "            \n",
        "        res = driver.page_source.encode('utf-8')\n",
        "        soup = BeautifulSoup(res, \"html.parser\")\n",
        "        data = soup.find(attrs={\"data-testid\": \"primaryColumn\"}).find_all(attrs={\"data-testid\": \"UserCell\"})\n",
        "\n",
        "        for counter in range(len(data)):\n",
        "            user = data[counter].find_all(\"span\")\n",
        "\n",
        "            full_name = user[1].text\n",
        "            handle = user[2].text\n",
        "\n",
        "            # not all users have a bio \n",
        "            try: \n",
        "                bio = user[5].text\n",
        "            except: \n",
        "                bio = None\n",
        "\n",
        "            # avoid duplicates\n",
        "            user_data = {\"full_name\": full_name, \n",
        "                          \"handle\": handle,\n",
        "                          \"bio\": bio\n",
        "                         }\n",
        "                \n",
        "            if user_data not in users: \n",
        "                users.append(user_data)\n",
        "\n",
        "        driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')       \n",
        "        last_height = current_height\n",
        "        sleep(5)\n",
        "        \n",
        "    df = pd.DataFrame(users)\n",
        "    df.to_csv(\"followings_dynamic.csv\", index=False)\n",
        "              \n",
        "# don't forget to manually login to your Twitter account to access the following list before proceeding (otherwise it will not load the list of followers!)\n",
        "driver.get(\"https://twitter.com/TilburgU/following\") \n",
        "\n",
        "# add a short pause here, or Twitter recognizes that it's a scraper and will block your request!\n",
        "sleep(1)\n",
        "\n",
        "twitter_followings_while()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BN-3kyZf-Q-Q"
      },
      "source": [
        "--- \n",
        "### 1.5 Search Tweets\n",
        "\n",
        "**Importance**  \n",
        "Although scraping data from a specific Twitter account is a good starting point, you may come across a scenario in which the tweets you're after come from a variety of sources. We can use Twitter's advanced search functionality to filter down on those tweets we're looking for.   \n",
        "\n",
        "On Twitter, you find a search bar at the top which allows you to search for a topic. In the right sidebar a search filter panel appears that includes `Advanced search` tools. Filling out any of the search boxes will automatically update the user input in the search box above. For example, tweets that contain either `cats` or `dogs` (or both) can be obtained with the search query `(cats OR dogs)`. The specified search term may occur in the user name, handle, bio, tweet, or in any of the replies of a thread. All search queries are case insensitive, so `cats` and `Cats` is considered equal."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t-fv5x_H-Q-R"
      },
      "source": [
        "<img src=\"https://github.com/hannesdatta/course-odcm/blob/dev/content/docs/tutorials/webscrapingadvanced/images/cats_dogs.gif?raw=true\" align=\"left\" width=60%/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vu51Yf7J-Q-R"
      },
      "source": [
        "Did you notice the URL was updated in accordance with our search parameters (remember from [icanhazdadjoke](https://icanhazdadjoke.com/api))? In fact, the new URL became: `https://twitter.com/search?q=(cats%20OR%20dogs)&src=typed_query`: \n",
        "* `q=` stands for search query\n",
        "* `(cats%20OR%20dogs)` corresponds with the contents of the query `(cats OR dogs)`\n",
        "* `&src=typed_query` indicates that we filled out the search query manually\n",
        "\n",
        "Note that `%20` is a space character. A full list of search commands and syntax can be found below: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EeEgXdx2-Q-R"
      },
      "source": [
        "| Command | Syntax | Interpretation | URL suffix | \n",
        "| :------- | :------- | :------ | :----- | \n",
        "| All of these words | `cats dogs` | Contains both `cats` and `dogs` | `cats%20dogs` | \n",
        "| Exact phrase | `\"cats\"` | Contains the exact phrase `cats` | `%22cats%22` | \n",
        "| Any of these words | `(cats OR dogs)` | Contains either `cats` or `dogs` (or both) | `(cats%20OR%20dogs)` | \n",
        "| None of these words | `-cats` | Does not contain `cats` | `-cats` | \n",
        "| These hashtags | `(#cats)` | Contains the hashtag `#cats` | `(%23cats)` | \n",
        "| Language | `lang:nl` | Tweets in specified language (default: all) | `lang%3Anl` | \n",
        "| From these accounts | `(from:hannesdatta)` | Tweets from Hannes Datta | `(from%3Ahannesdatta)` | \n",
        "| Replies to these accounts | `(to:hannesdatta)` | Replies to tweets from Hannes Datta | `(to%3Ahannesdatta)` | \n",
        "| Mentioning these accounts | `(@hannesdatta)` | Tweets mentioning Hannes Datta | `(%40hannesdatta)` | \n",
        "| Minimum replies | `min_replies:10` | Tweets with at least 10 replies | `min_replies%3A10` | \n",
        "| Minimum likes | `min_faves:10` | Tweets with at least 10 likes | `min_faves%3A10` | \n",
        "| Minimum retweets | `min_retweets:10` | Tweets with at least 10 retweets | `min_retweets%3A10` | \n",
        "| From (date) | `since:01-01-2020` | Tweets after the 1st of January 2020 | `since%3A01-01-2020` | \n",
        "| To (date) | `until:01-01-2020` | Tweets before the 1st of January 2020 | `until%3A01-01-2020` | "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6RhA1Oa-Q-R"
      },
      "source": [
        "**Let's try it out!**  \n",
        "Play around with the filters until you get the hang of it. How do you chain multiple search commands? (e.g., tweets about cats with at least 100 likes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XwwTq-1f-Q-S"
      },
      "source": [
        "**Exercise 5**  \n",
        "1. Suppose that you're responsible for Tilburg University's Public Relations & Digital Communication and want to keep an eye on what others are writing about the university. Compile a search query in the Twitter web interface to collect tweets that refer to either `Tilburg University` (English) or `Tilburg Universiteit` (Dutch). Take note of the search command and keep track of the URL.\n",
        "2. Write a function `search_tweets()` that takes a Twitter search query as input and returns a list of all tweets (including a link to the original tweet so that your PR colleague can respond if necessary) as a csv-file. Test your function with the URL of the previous question. \n",
        "3. After further inspection, you come to the conclusion that your data include a dozen or so train (Dutch: \"trein\") disruption alerts related to \"Tilburg Universiteit\" (e.g., see example below). How can you easily exclude those tweets upfront? \n",
        "\n",
        "<img src=\"https://github.com/hannesdatta/course-odcm/blob/dev/content/docs/tutorials/webscrapingadvanced/images/train_alerts.png?raw=true![image.png](attachment:image.png)\" align=\"left\" width=40%/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qpTkmf71-Q-S"
      },
      "source": [
        "*English: #NS Resolved: Tilburg University-Eindhoven: broken train. Boxtel-Eindhoven C.: train traffic has resumed.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gSPYwy1o-Q-S"
      },
      "outputs": [],
      "source": [
        "# Question 1 and 2\n",
        "def search_tweets(query):\n",
        "    driver.get(\"https://twitter.com/search?q=\" + query) \n",
        "    tweets = []\n",
        "    last_height = 0\n",
        "    current_height = 0\n",
        "\n",
        "    iterator = 0\n",
        "    while True: \n",
        "        iterator += 1\n",
        "        \n",
        "        current_height = driver.execute_script('return document.body.scrollHeight')\n",
        "\n",
        "        if current_height == last_height: \n",
        "            break\n",
        "\n",
        "        res = driver.page_source.encode('utf-8')\n",
        "        soup = BeautifulSoup(res, \"html.parser\")\n",
        "        data = soup.find(attrs={\"data-testid\": \"primaryColumn\"}).find_all(attrs={\"data-testid\":\"tweet\"})\n",
        "\n",
        "        for counter in range(len(data)):\n",
        "            tweet = data[counter].find_all(attrs={\"dir\":\"auto\"})\n",
        "\n",
        "            text = tweet[4].text\n",
        "            try:\n",
        "                link = tweet[3]['href']\n",
        "            except:\n",
        "                link = '/LINK-NOT-AVAILABLE'\n",
        "\n",
        "            tweet_data = {\n",
        "                            \"tweet\": text, \n",
        "                            \"link\": \"https://twitter.com/\" + link\n",
        "                         }\n",
        "\n",
        "            if tweet_data not in tweets: \n",
        "                tweets.append(tweet_data)\n",
        "\n",
        "        driver.execute_script('window.scrollTo(0, document.body.scrollHeight);')       \n",
        "        last_height = current_height\n",
        "        sleep(5)\n",
        "        if (iterator == 2): break\n",
        "        \n",
        "    df = pd.DataFrame(tweets)\n",
        "    df.to_csv(\"search_tweets.csv\", index=False)\n",
        "\n",
        "collected_tweets = search_tweets('(\"Tilburg University\" OR \"Tilburg Universiteit\")')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RtTHHAF7-Q-S"
      },
      "outputs": [],
      "source": [
        "# Question 3\n",
        "\n",
        "# add \"-trein\" to the URL - see table above (as simple as that!)\n",
        "search_tweets('\"Tilburg University\" OR \"Tilburg Universiteit\" -\"trein\"')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pMaGMg2s-Q-T"
      },
      "source": [
        "By now, you have undoubtedly noticed that by default Twitter search shows the so-called \"Top\" tweets. However, you can click on any of the other tabs so that it returns data from a specific type, data sorted in a given order, or from Twitter users that meet your selection. Note that you can select up to 1 tab and 1 search filter. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d0U8OzAs-Q-T"
      },
      "source": [
        "| Tab | Interpretation | URL suffix | \n",
        "| :----- | :------ | :------ |\n",
        "| Latest | Sort the tweets chronologically (in descending order) | `&-f=live` |\n",
        "| People | Return a list of Twitter accounts (as opposed to tweets) | `&-f=user` |\n",
        "| Photos | Filter on tweets that include an image | `&-f=image` |\n",
        "| Videos | Filter on tweets that include a video | `&-f=video` |\n",
        "\n",
        "| Search filter| Interpretation | URL suffix | \n",
        "| :----- | :------ | :------ |\n",
        "| Near you | Return tweets that are published by someone in your neighborhood | `&lf=on` |\n",
        "| From people you follow | Filter on tweets from accounts you follow | `&pf=on` |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KVkjA59r-Q-T"
      },
      "source": [
        "**Exercise 6**  \n",
        "Brainstorm about search strategies you can deploy to narrow down your Twitter search and make the output of Exercise 5 more useful and actionable for the PR department. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "thCdcf6t-Q-T"
      },
      "source": [
        "**Your answer**  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AVh2p4ZQ-Q-T"
      },
      "source": [
        "... (double click to edit this cell)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jul1_nNR-Q-U"
      },
      "source": [
        "**Solution**  \n",
        "You may want to filter down on the latest tweets near Tilburg to separate the signal the noise since. Some public statements may require immediate action after all.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d6GGJA9_-Q-U"
      },
      "source": [
        "---\n",
        "### 1.6 Wrap-Up\n",
        "In this section, you have learned how to use a browser emulation (Selenium) to access data that is hidden behind a login-screen or only appears once scrolling down. This opens up a whole new world of opportunities for you to explore. Some ideas to explore on your own: analyze the hashtags that get the highest engagement, plot the Twitter follower growth over time, identify upcoming influencers, and conduct market research to derive insights to grow your audience. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NAOSJy4v-Q-U"
      },
      "source": [
        "---\n",
        "## 2. Instagram"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A6_JFjM7-Q-U"
      },
      "source": [
        "### 2.1 Click Sites Programmatically"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qa6gG8qs-Q-U"
      },
      "source": [
        "**Importance**  \n",
        "Now that you got a feeling for how to work with Selenium, we're going to have a brief look at how to scrape data from Instagram. In many ways, the techniques and code follow the same logic as above. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hE4cUVKc-Q-V"
      },
      "source": [
        "First, we navigate towards the [Instagram account](https://www.instagram.com/tilburguniversity/) of Tilburg University with Selenium, accept the cookies, login with our own credentials (we recommend creating a separate account for scraping purposes), and close any windows that may occur (e.g. \"Save your login info\", or \"Turn on notifications\")."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0DP-oB2v-Q-V"
      },
      "outputs": [],
      "source": [
        "# first create a new driver object if you accicentally closed it (see beginning of this notebook)\n",
        "driver.get(\"https://www.instagram.com/tilburguniversity/\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5UW0pH2-Q-V"
      },
      "source": [
        "The page structure of Instagram is slightly different from Twitter in the sense that it presents an overview of all images from which we gather the links to all separate posts (like the webshop example). Inspection of the HTML structure tells us that a link to each post can be obtained from the `<a>` tags with attribute `tabindex:\"0\"` within the `<article>` tags: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t8cRkg_B-Q-V"
      },
      "outputs": [],
      "source": [
        "# login and close pop-ups before proceeding!\n",
        "driver.get(\"https://www.instagram.com/tilburguniversity/\")\n",
        "res = driver.page_source.encode('utf-8')\n",
        "soup = BeautifulSoup(res, \"html.parser\")\n",
        "link = soup.find(\"article\").find_all(attrs={\"tabindex\": \"0\"})[0][\"href\"]\n",
        "print(f\"The link of the most recent post is: http://www.instagram.com/{link}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "40_VLDHx-Q-V"
      },
      "source": [
        "**Let's try it out!**  \n",
        "Get the links from all other posts! How many images/videos can you access (without scrolling down)? What does the `alt` attribute in the `img` tags tell you? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LGFBzk4--Q-W"
      },
      "source": [
        "---\n",
        "One of the [links](https://www.instagram.com/p/B_KalD_DHcV/) that you'll get this way, showcases a picture of the Heuvelplein in Tilburg. Although comments are listed by default, replies are only visible after clicking on `View replies`. Note that replies are not simply hidden somewhere hidden in the code, in fact, a new `<div>` block of code is added to the source code (see below).\n",
        "\n",
        "<img src=\"https://github.com/hannesdatta/course-odcm/blob/dev/content/docs/tutorials/webscrapingadvanced/images/click_instagram.gif?raw=true\" align=\"left\" width=80%/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wISONL9W-Q-W"
      },
      "source": [
        "Sure, you can click on each link manually but that would take ages for posts with many replies. Hence, we seek for the class of the `View replies` element and use Selenium to *click* on the link for us. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s1QSwH2J-Q-W"
      },
      "outputs": [],
      "source": [
        "driver.get(\"https://www.instagram.com/p/B_KalD_DHcV/\")\n",
        "view_replies = driver.find_elements_by_class_name(\"EizgU\")[0]\n",
        "sleep(3) # wait a few seconds before the page is fully loaded\n",
        "view_replies.click()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vlR5A5ej-Q-W"
      },
      "source": [
        "**Let's try it out!**  \n",
        "Run the `view_replies.click()` command one more time (without the other lines of code), what happens then? Which replies are triggered by this click? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tz2hj7-Q-Q-W"
      },
      "source": [
        "**Exercise 6**  \n",
        "Write a program that extracts all replies for a given Instagram URL and writes it to a csv file. It should include both the username of the author and the reply text (not the original comments). You can break down your code in two separate functions: `click_buttons()` and `extract_replies()`. \n",
        "\n",
        "* `click_buttons()` should identify the location of the button class (e.g., whether the contents contains the text \"View replies\") and click on all those buttons. Ideally, your function should work without hardcoding a class name of the button as we did in the previous example. \n",
        "\n",
        "\n",
        "* `extract_replies()` scrapes all replies and usernames once the buttons are clicked upon. Keep in mind that a comment may have more than one reply. Tip: you can use `.find_parent()` to move up one level in the Document Object Model (DOM) hierarchy.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0PtnOo_z-Q-W"
      },
      "outputs": [],
      "source": [
        "# your answer goes here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DKwIDr8F-Q-X"
      },
      "outputs": [],
      "source": [
        "# solution\n",
        "def click_buttons(url):\n",
        "    driver.get(url)\n",
        "    \n",
        "    # extract HTML of the Instagram page\n",
        "    res = driver.page_source.encode('utf-8')\n",
        "    soup = BeautifulSoup(res, \"html.parser\")\n",
        "\n",
        "    # there are a bunch of buttons on the page, but the one we're after has the text \"View replies\" on it\n",
        "    temp = []\n",
        "    for element in soup.find_all(\"button\"): \n",
        "        if \"View replies\" in element.text: \n",
        "            temp.append(element)\n",
        "\n",
        "    # all buttons have the same class so it doesn't really matter which one we pick        \n",
        "    class_name = temp[0].find(\"span\")[\"class\"][0] \n",
        "    \n",
        "    # click on buttons for the class we identified above\n",
        "    buttons = driver.find_elements_by_class_name(class_name)\n",
        "    for button in buttons: \n",
        "        button.click()\n",
        "        sleep(1)\n",
        "        \n",
        "    return class_name\n",
        "\n",
        "\n",
        "def extract_replies(url):\n",
        "    replies = []\n",
        "    class_name = click_buttons(url)  # here we call the function above which clicks on all buttons and returns the class_name (of the buttons) which we re-use below\n",
        "    \n",
        "    # extract HTML again (but this time with the replies included)\n",
        "    res = driver.page_source.encode('utf-8')\n",
        "    soup = BeautifulSoup(res, \"html.parser\")\n",
        "    data = soup.find_all(class_ = class_name)\n",
        "\n",
        "    # collect replies\n",
        "    for comment in data: \n",
        "        for reply in comment.find_parent(\"ul\").find_all(class_ = \"notranslate\"):\n",
        "            text = reply.find_parent(\"span\").text\n",
        "            user = reply.find_parent(\"div\").find(attrs={\"tabindex\": 0}).text\n",
        "            replies.append({\"reply\": text, \n",
        "                            \"user\": user})\n",
        "\n",
        "    df = pd.DataFrame(replies)\n",
        "    df.to_csv(\"replies.csv\", index=False)\n",
        "    \n",
        "extract_replies(\"https://www.instagram.com/p/B_KalD_DHcV/\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z3GDi8cz-Q-X"
      },
      "source": [
        "---\n",
        "### 2.2 Scrape Image Files"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y6OFH8Bx-Q-X"
      },
      "source": [
        "**Importance**  \n",
        "In previous examples, we looked at scraping textual data from a web page. On Instagram, however, it would make sense to store the image files as well. To this end, we extract a link to the image source (`image_link`) and pass it to the `wget` library. You can name the image whatever you want (e.g., `my_image.jpg`). By default, the image is stored in your current working directory (i.e., where this notebook resides).\n",
        "\n",
        "Please note that the `wget` module is not a standard Python package, so before running the cell below you need to install it first. Either type `pip install wget` in your terminal or use the Anaconda Navigator interface (see below - in your case there will likely be only the base (root) environment)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rreXV17e-Q-X"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/hannesdatta/course-odcm/master/content/docs/tutorials/webscrapingadvanced/images/install_packages.gif\" align=\"left\" width=70%/>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fHb9UxcC-Q-Y"
      },
      "outputs": [],
      "source": [
        "!pip install wget\n",
        "import wget\n",
        "image_link = soup.find(attrs={\"decoding\": \"auto\"})[\"src\"]\n",
        "wget.download(image_link, \"my_image.jpg\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zTtkcWgC-Q-Y"
      },
      "source": [
        "**Let's try it out!**  \n",
        "Visit another Instagram page (with `drive.get()`), extract the HTML of that page, recreate a `soup` object, and see whether you can store the image on your local machine. Can you also scrape videos in this way? How about Instagram [carousels](https://www.instagram.com/p/CIGLXWMoPkh/) (i.e., posts that contain multiple media)? "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CRQQ7Drw-Q-Y"
      },
      "source": [
        "**Exercise 7**  \n",
        "Write a function `scrape_image()` that takes an Instagram URL and returns the images associated with the post. \n",
        "\n",
        "* Use the post identifier (e.g., for [this](https://www.instagram.com/p/CIGLXWMoPkh/) post use `CIGLXWMoPkh`) appended by `_image[NR].jpg` (e.g., `CIGLXWMoPkh_image1.jpg`, `CIGLXWMoPkh_image2.jpg`, etc.) as the file name for the images. \n",
        "* For this exercise, you can restrict yourself to the first two images of the image carousel (more on that later).\n",
        "* Skip the posts displayed under \"More posts from ...\" (see below):\n",
        "\n",
        "![](https://raw.githubusercontent.com/hannesdatta/course-odcm/master/content/docs/tutorials/webscrapingadvanced/images/more_posts.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8NyyAuv0-Q-Y"
      },
      "outputs": [],
      "source": [
        "# your answer goes here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Oxh2_diD-Q-Y"
      },
      "outputs": [],
      "source": [
        "# solution\n",
        "def scrape_image(url):\n",
        "    driver.get(url)\n",
        "   \n",
        "    res = driver.page_source.encode('utf-8')\n",
        "    soup = BeautifulSoup(res, \"html.parser\")\n",
        "    \n",
        "    data = soup.find(attrs={\"role\": \"presentation\"}).find_all(attrs={\"decoding\": \"auto\"})\n",
        "    \n",
        "    for counter in range(len(data)):\n",
        "        item_link = data[counter][\"src\"]\n",
        "        file_name = url.strip('/')[-11:] + \"_image\" + str(counter + 1) + \".jpg\"\n",
        "        wget.download(item_link, file_name)\n",
        "        \n",
        "scrape_image(\"https://www.instagram.com/p/CIGLXWMoPkh/\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FM1ymNNW-Q-Z"
      },
      "source": [
        "**Exercise 8**  \n",
        "By default, Instagram only loads the first two images of a image carousel. Therefore, if you run `scrape_image()` above it will export two out of three images. If you click on the arrow pointing to the right, however, the 3rd image is loaded and becomes extractable in the code (see below). Extend the functionality of `scrape_image()` so that it scrapes all images. Your code should still work for a \"normal\" image (not a carousel). \n",
        "\n",
        "![](https://raw.githubusercontent.com/hannesdatta/course-odcm/master/content/docs/tutorials/webscrapingadvanced/images/load_images.gif)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HghuN4gf-Q-Z"
      },
      "outputs": [],
      "source": [
        "# your answer goes here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Gu8Adwj9-Q-Z"
      },
      "outputs": [],
      "source": [
        "# solution\n",
        "def scrape_image(url):\n",
        "    res = driver.page_source.encode('utf-8')\n",
        "    soup = BeautifulSoup(res, \"html.parser\")\n",
        "    \n",
        "    data = soup.find(attrs={\"role\": \"presentation\"}).find_all(attrs={\"decoding\": \"auto\"})\n",
        "    \n",
        "    for counter in range(len(data)):\n",
        "        item_link = data[counter][\"src\"]\n",
        "        file_name = url.strip('/')[-11:] + \"_image\" + str(counter + 1) + \".jpg\"\n",
        "        wget.download(item_link, file_name)\n",
        "\n",
        "def right_arrow():\n",
        "    return driver.find_elements_by_class_name(\"coreSpriteRightChevron\")  # this is the class for the right arrow button\n",
        "\n",
        "def scrape_all_images(url):\n",
        "    driver.get(url)\n",
        "    arrow = right_arrow()\n",
        "\n",
        "    # click on right arrow until you reached the last image (then there is no right arrow class anymore)\n",
        "    while len(arrow) > 0: \n",
        "        arrow[0].click()\n",
        "        arrow = right_arrow()\n",
        "    \n",
        "    # scrape images\n",
        "    scrape_image(url)\n",
        "\n",
        "scrape_all_images(\"https://www.instagram.com/p/CIGLXWMoPkh/\") # image carousel\n",
        "scrape_all_images(\"https://www.instagram.com/p/CJItoJODy9y/\") # normal image"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZF0zbjnq-Q-Z"
      },
      "source": [
        "---\n",
        "### 2.3 Wrap-Up\n",
        "We have gone a long way since the Web data for dummies tutorial and learned a variety of techniques to get to the data we're after. In particular, social network services rely on scrolling and clicking interactions to achieve this. As said many times before, this is just the beginning of your scraping journey. For example, see whether you can exploit Instagram's explore function to filter down on posts with a specific [tag](https://www.instagram.com/explore/tags/tilburguniversity/)\n",
        "or [location](https://www.instagram.com/explore/locations/213125596/tilburg-netherlands/). Good luck!\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "webscraping-advanced.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "oEEn7ELe-Q-L",
        "BN-3kyZf-Q-Q",
        "d6GGJA9_-Q-U",
        "A6_JFjM7-Q-U",
        "z3GDi8cz-Q-X",
        "ZF0zbjnq-Q-Z"
      ],
      "include_colab_link": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}