Wow, thanks for all the tips, everyone! I’ve been trying out Scrapy-rotating-proxy, and it’s been a huge help with IP rotation. Also, scrapy-fake-useragent is a gem—headers are way less of a headache now.
I’m still on the fence about paid proxies, but after reading your suggestions, I think I’ll give Bright Data or Oxylabs a shot.
One quick follow-up: do you guys use any specific settings for handling CAPTCHAs, or do you just rely on proxies and delays to avoid them?
Cheers again for all the advice!
I’m still on the fence about paid proxies, but after reading your suggestions, I think I’ll give Bright Data or Oxylabs a shot.
One quick follow-up: do you guys use any specific settings for handling CAPTCHAs, or do you just rely on proxies and delays to avoid them?
Cheers again for all the advice!