What's the best way to build a Python web crawler for scraping large sites? or How do I optimize my P

18 Replies, 901 Views

BS4 is beginner-friendly, but Scrapy is built for scale. If you’re doing a python web crawler for product data, Scrapy’s item pipelines will keep you sane.

The curve isn’t *that* bad—just follow a tutorial or two.

And hey, if you hate it, you can always fall back to BS4. But you’ll probably outgrow it fast.

Messages In This Thread



Users browsing this thread: 1 Guest(s)