Hey! Using proxy with scrapy is all about trial and error.
I’d recommend rotating proxies every 10 requests and setting a delay of at least 2 seconds. Also, use a middleware like scrapy-user-agents to randomize your user-agent.
For paid proxies, I use Oxylabs. It’s pricey, but it’s worth it for the reliability.
I’d recommend rotating proxies every 10 requests and setting a delay of at least 2 seconds. Also, use a middleware like scrapy-user-agents to randomize your user-agent.
For paid proxies, I use Oxylabs. It’s pricey, but it’s worth it for the reliability.