What's the best way to build a Python web crawler for scraping large sites? or How can I optimize my

23 Replies, 2064 Views

Lol @ the robots.txt fail. Been there!

For error handling, use `try-except` blocks *everywhere*. Network errors, 404s, weird encoding—cover your bases.

And cache responses! `requests-cache` saves so much time if you’re re-running your python web crawler.

Messages In This Thread



Users browsing this thread: 1 Guest(s)