What’s the best way to build a Python web crawler for scraping large sites? Alternatively: How can I

Wow, thanks for all the tips! Didn’t expect so many replies.

I’ll give Scrapy another shot with rotating proxies (probably Smartproxy) and throttle the requests.

Quick Q: Anyone tried Scrapy Cloud? Is it worth it for handling bans automatically, or should I just stick to local tweaking?

Also, big shoutout for the proxy recommendations—saved me hours of trial and error. 🙌