Hey! Using proxy with scrapy can be frustrating, but it’s all about the setup.
I’d suggest using Scrapy-proxy-pool for IP rotation. It’s easy to integrate and works like a charm.
For headers, I use scrapy-user-agents. It’s a must to avoid detection.
Delays? I keep mine between 2-4 seconds. It’s a sweet spot.
And yeah, free proxies are a waste of time. I use GeoSurf, and it’s been great.
Good luck!
I’d suggest using Scrapy-proxy-pool for IP rotation. It’s easy to integrate and works like a charm.
For headers, I use scrapy-user-agents. It’s a must to avoid detection.
Delays? I keep mine between 2-4 seconds. It’s a sweet spot.
And yeah, free proxies are a waste of time. I use GeoSurf, and it’s been great.
Good luck!