Wow, thanks so much for all the suggestions! I didn’t expect so many helpful replies.
I’ve already tried Scrapy-Rotating-Proxies based on one of the comments, and it’s working way better than what I was doing before. Still getting a few blocks, but I’ll tweak the delays and user-agents like you all suggested.
Quick question: anyone know if there’s a way to handle CAPTCHAs automatically? I’m hitting a few sites that throw them at me, and it’s slowing things down.
Thanks again, you guys are awesome!
I’ve already tried Scrapy-Rotating-Proxies based on one of the comments, and it’s working way better than what I was doing before. Still getting a few blocks, but I’ll tweak the delays and user-agents like you all suggested.
Quick question: anyone know if there’s a way to handle CAPTCHAs automatically? I’m hitting a few sites that throw them at me, and it’s slowing things down.
Thanks again, you guys are awesome!