Part 0: Setup

Initial setup of your scraper environment.

IMPORTANT: Please be sure to work through the machine setup before proceeding.
Need to cheat a bit? The complete scraper can be found in new-coder/scrape.


You’ll neet PostgreSQL installed:

Project Setup

Within your terminal:

Change into the Web Scraping project.

$ cd new-coder/scrape

Make sure you’ve installed virtualenvwrapper and followed the steps from the machine setup to set up your terminal correctly.

Make a new virtual environment specific to your scrape project project:

$ mkvirtualenv ScrapeProj

You should see (ScrapeProj) before your prompt. Now install package requirements with the following command for this project.

(ScrapeProj) $ pip install -r requirements.txt

Your virtual environment will store the required packages in a self-contained area to not mess up with other Python projects.

Lastly, navigate to your project workspace that you setup earlier from the machine setup:

(ScrapeProj) $ cd ..
(ScrapeProj) $ cd scrape_workspace

Let’s continue!