What's the best way to build a Python web crawler for scraping large sites? or How can I optimize my

18 Replies, 2168 Views

OP here—thanks for all the tips! Gonna try Scrapy with rotating proxies and see how it goes.

Quick Q: Anyone got a good tutorial for setting up Scrapy with Redis? Found a few but not sure which one’s up-to-date.

Also, shoutout to the person who suggested checking for APIs—totally saved me from overcomplicating it lol.

Messages In This Thread



Users browsing this thread: 1 Guest(s)