What’s the best way to build a Python web crawler for scraping large sites? Alternatively: How can I

Honestly, for a python web crawler, start simple. Use Requests + BeautifulSoup with a *slow* delay (3+ sec) and see if that avoids bans.

If you *must* scale, Scrapy is the way. But yeah, the learning curve sucks.

Proxies: Luminati is gold but $$$. For free, try scraping public proxy lists (ironic, I know).

Async? Only if you’re ready for the rabbit hole.