-
Notifications
You must be signed in to change notification settings - Fork 8
Parsing Example 2
###Overview This example will walk you through how to parse a website using Selenium. Selenium allows you to open a webpage and automate the process of clicking through a website and extracting information. Selenium is the preferred choice over Beautiful Soup when parsing a website that requires navigating through menus or loading new web pages. Basically if the website requires you to click elements on the page or enter information in order to get to the information you want, you should use Selenium. Parsing a website will require you to be at least a little familiar with HTML so brush up on that here if you aren't comfortable reading HTML.
###Step 1: Get a Feel for the Website I always start by clicking through the website and making a mental note of the structure. Where do I want to start the process? How do I get from there to all the cameras I need to get to? Once I'm on a page that has cameras to parse, can I get to another page of cameras from there or do I have to go back to the start point? Come up with a plan as to how you want to navigate the site. Once you have that figured out you are ready to start coding.
###Step 2: Navigating the Website For this example we will be parsing the 511 Alberta Site. This site has a dropdown menu to select different highways.  Each option will have a different set of cameras to choose from. Then you need to click each camera box to pull up a new page that contains different camera angles for the chosen camera. Each camera page looks something like this: 
With alternate angles on the left hand side or something like this, with alternate angles on the bottom:

©️ 2016 Cam2 Research Group