Part 0: Setup

Initial setup of your scraper environment.

Postgres

You’ll neet PostgreSQL installed:

• Ubuntu: https://help.ubuntu.com/community/PostgreSQL (install both client & server)
• Debian: https://wiki.debian.org/PostgreSql#Installation (install both client & server)
• Fedora/RHEL/CentOS/Scientific: http://www.postgresql.org/download/linux/redhat/ (install both client & server)
• Windows: http://www.postgresql.org/download/windows/
• Mac: http://postgresapp.com/

Project Setup

Within your terminal:

Change into the Web Scraping project.

1

$ cd new-coder/scrape

Make sure you’ve installed virtualenvwrapper and followed the steps from the machine setup to set up your terminal correctly.

Make a new virtual environment specific to your scrape project project:

1

$ mkvirtualenv ScrapeProj

You should see (ScrapeProj) before your prompt. Now install package requirements with the following command for this project.

1

(ScrapeProj) $ pip install -r requirements.txt

Your virtual environment will store the required packages in a self-contained area to not mess up with other Python projects.

Lastly, navigate to your project workspace that you setup earlier from the machine setup:

1
2

(ScrapeProj) $ cd ..
(ScrapeProj) $ cd scrape_workspace

Let’s continue!