What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl

Yo, for optimizing your web crawler python, async is a game-changer! aiohttp + asyncio speeds things up massively.

Also, set delays between 2-5 secs to avoid bans.

Proxies? Try free ones from https://free-proxy-list.net/, but paid ones like Luminati are way more reliable.