What's the best way to build a Python web crawler for scraping large sites? or How do I optimize my P

If you’re worried about Scrapy being too complex, try `requests-html`. It’s like BS4 but with JS support and a simpler async setup.

But yeah, for *serious* scraping, Scrapy is king. The middleware system alone makes it worth the hassle.

Also, check out `scrapy-splash` if you need to render JS.