What's the best way to start crawling with Python for web scraping? or How can I improve my crawling

Biggest pitfall? Not handling errors gracefully. Your script WILL fail, so plan for it.

For crawling with Python, start small—scrape one page perfectly before scaling.

Proxies? Only if you’re doing heavy scraping. Otherwise, just throttle and rotate user-agents.