Scrapy Proxy Middleware — Rotating Residential IPs per Request

A drop-in Scrapy middleware that routes every request through a rotating residential proxy, with optional per-request sticky sessions and country override.

Quick start

1. Copy middlewares.py into your Scrapy project.
2. Enable it in settings.py:

# settings.py
QP_USER = "YOUR_USERNAME"
QP_PASS = "YOUR_PASSWORD"
QP_COUNTRY = "us"  # default exit country, override per request via meta

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ResidentialProxyMiddleware": 350,
}

# Be a good citizen + keep block rates low
CONCURRENT_REQUESTS = 8
DOWNLOAD_TIMEOUT = 40
RETRY_TIMES = 3

3. That's it — every request now exits from a fresh residential IP.

Per-request control

# force a German exit for one request
yield scrapy.Request(url, meta={"qp_country": "de"})

# pin a sticky session for a login flow (same IP across the sequence)
yield scrapy.FormRequest(login_url, formdata=creds, meta={"qp_session": "job42"})

The middleware maps qp_session to the sticky port (10000) with a 10-minute lifetime, and everything else to the rotating port (9000).

Why a middleware instead of meta={"proxy": ...} everywhere

Centralizing proxy logic means retries automatically get a new IP, sessions are consistent, and spiders stay clean. Combined with RETRY_TIMES, a blocked response simply retries through a different residential exit.

Proxy backend

Built for QuantumProxies rotating proxies — per-GB residential bandwidth in 195+ countries with city/ISP targeting. Credentials come from the dashboard; the gateway is residentialboson.quantumproxies.io (rotating :9000, sticky :10000, SOCKS5 :12000).