Hey! Using proxy with scrapy can be a bit of a headache, but it’s worth it. One tip: always use HTTPS proxies if you’re scraping secure sites. HTTP proxies can sometimes fail or get blocked.
For rotating proxies, I’d recommend checking out the `scrapy-proxies` library. It’s pretty straightforward to set up and works well for most use cases.
Also, if you’re dealing with IP bans, try adding a retry middleware. It’ll automatically retry failed requests with a different proxy.
For rotating proxies, I’d recommend checking out the `scrapy-proxies` library. It’s pretty straightforward to set up and works well for most use cases.
Also, if you’re dealing with IP bans, try adding a retry middleware. It’ll automatically retry failed requests with a different proxy.