Yo, been there! Using proxy with scrapy can be a nightmare if you don’t set it up right.
First, ditch free proxies. They’re slow and get blocked super fast. I use Oxylabs—kinda pricey but worth it.
For IP rotation, try setting up a proxy pool with a service like ProxyMesh. It’s easy to integrate with scrapy.
Headers are key too. Use scrapy-fake-useragent to randomize them. And yeah, random delays help, but don’t overdo it—2-5 seconds is usually enough.
Hope that helps!
First, ditch free proxies. They’re slow and get blocked super fast. I use Oxylabs—kinda pricey but worth it.
For IP rotation, try setting up a proxy pool with a service like ProxyMesh. It’s easy to integrate with scrapy.
Headers are key too. Use scrapy-fake-useragent to randomize them. And yeah, random delays help, but don’t overdo it—2-5 seconds is usually enough.
Hope that helps!