[b]"What’s the best way to build a Python web crawler for scraping large sites?"[/b] Alternatively: [b]"How can I

If you’re scraping *massive* sites, Scrapy + Redis is a powerhouse combo. Lets you distribute crawling across machines.

Proxies are a must—I use GeoSurf. And yeah, randomize user-agents (fake-useragent lib is handy).

Async is cool, but Scrapy’s already async under the hood. Stick with it and tweak settings like CONCURRENT_REQUESTS.

P.S. Check out Scrapy’s AutoThrottle feature to avoid bans.