Hey there! Using proxy with scrapy is all about balance.
I’d suggest using Scrapy-UserAgents alongside your proxies. It randomizes user-agents, which helps avoid detection.
For proxy rotation, I’ve had good results with Zyte (formerly Scrapinghub). They offer a proxy service specifically for scraping, and it’s pretty solid.
Also, keep an eye on your request headers. Some sites block requests if they look suspicious.
Hope this helps!
I’d suggest using Scrapy-UserAgents alongside your proxies. It randomizes user-agents, which helps avoid detection.
For proxy rotation, I’ve had good results with Zyte (formerly Scrapinghub). They offer a proxy service specifically for scraping, and it’s pretty solid.
Also, keep an eye on your request headers. Some sites block requests if they look suspicious.
Hope this helps!