[b]"What's the best way to build a Python web crawler for scraping large sites?"[/b] or [b]"How can I optimize my

Scrapy is def the way to go for large-scale scraping. It’s built for handling big sites and has built-in stuff like throttling and concurrent requests.

For rate limits, use rotating proxies (check out ScraperAPI or Luminati) and randomize your user-agent. Also, set a polite DOWNLOAD_DELAY in settings.py.

If you’re scraping *really* big, maybe look into distributed crawling with Scrapy + Redis.