What's the best way to build a Python web crawler for scraping large sites? or How do I optimize my P

Scrapy’s learning curve is overhyped. It’s not *that* hard if you’ve got basic Python skills. The docs are solid, and the community’s helpful.

For a python web crawler handling product pages, Scrapy’s built-in exporters (JSON, CSV) will save you tons of time.

BS4 is fine, but you’ll end up reinventing the wheel for stuff Scrapy does out of the box.