Hey! Using proxy with scrapy can definitely be tricky. I’ve found that rotating proxies every 5-10 requests works best for me. Also, setting a random delay between 2-5 seconds helps avoid getting blocked.
As for free vs paid proxies, I’d say go for paid if you’re scraping at scale. Free ones are too unreliable. I use Bright Data—it’s a bit pricey but worth it for the reliability.
Good luck! �
As for free vs paid proxies, I’d say go for paid if you’re scraping at scale. Free ones are too unreliable. I use Bright Data—it’s a bit pricey but worth it for the reliability.
Good luck! �