Hey! Using proxy with scrapy can be a headache, but here’s what works for me:
- Rotate proxies every 10-15 requests.
- Set a random delay between 3-7 seconds.
- Use a mix of datacenter and residential proxies.
For tools, I recommend ProxyMesh or Smartproxy. They’re affordable and reliable.
Free proxies? Forget it. They’re more trouble than they’re worth.
- Rotate proxies every 10-15 requests.
- Set a random delay between 3-7 seconds.
- Use a mix of datacenter and residential proxies.
For tools, I recommend ProxyMesh or Smartproxy. They’re affordable and reliable.
Free proxies? Forget it. They’re more trouble than they’re worth.