"Web Scraping with Scrapy" Companion Code

This is the companion code for the article "Web Scraping with Scrapy" published in the March issue of Digit.

This code is best used as a reference when writing your own project. Avoid copy-pasting the code. Instead try to follow the instructions in the article and refer to the code if you feel something is not working as expected.

This project is organised in separate branches for each step taken in the article, with a few extra comments thrown in. You should see dropdown showing "Branch: master" in the GitHub UI that you can use to switch branches.

The default branch will show a freshly-created scrapy project, the result of running:

scrapy startproject github

Some useful resources:

• Python Download
• Scrapy Project Website
• Introduction to CSS Selectors
• XPath on MDN
• Python Regular Expressions