README.md

Proxy Port integration for Scrapy.

About Proxy Port

Prerequisites

To use this package you will need a free API key. Get your API key here. Detailed instructions here.

Installation

$ pip install scrapyproxyport

Getting Started

You need to assign an API key. This can be done either through an environment variable

$ export PROXY_PORT_API_KEY=<API_KEY>

or inside settings.py.

# inside <your_project>/settings.py

PROXY_PORT_API_KEY = '<API_KEY>'
DOWNLOADER_MIDDLEWARES = {
    # Add middleware with order number right before CookiesMiddleware
    'scrapyproxyport.middlewares.ProxyMiddleware': 898,
    'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 899,
}
DOWNLOAD_TIMEOUT = 10
RETRY_TIMES = 20
DUPEFILTER_CLASS = 'scrapy.dupefilters.BaseDupeFilter'

Example Scrapy project

scrapy-proxyport-example