[b]"What’s the best way to build a Python web crawler for scraping large sites?"[/b] Alternatively: [b]"How can I

"What’s the best way to build a python web crawler for scraping large sites?"

Hey folks! I’m trying to scrape a massive site (like, *thousands* of pages) and my current python web crawler is kinda slow + keeps getting blocked. 😅

I’ve messed around with Scrapy, but the learning curve’s steep. BeautifulSoup + Requests is easier, but idk if it scales well.

Any tips? Like:
- How to speed things up without getting banned?
- Best libs for handling big jobs?
- Should I even bother with async stuff?

Also, anyone got experience with rotating proxies or user-agents? My script keeps getting slapped by Cloudflare lol.

Thanks in advance! 🙏 (and sorry if this has been asked a million times)