What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl

22 Replies, 1221 Views

Scrapy’s still king for large-scale web crawler python projects in 2024. Middlewares, pipelines, and built-in concurrency make it unbeatable.

Alternatives? Maybe `httpx` + `parsel` for smaller tasks, but Scrapy’s worth the effort.

Messages In This Thread



Users browsing this thread: 1 Guest(s)