What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl

Wow, thanks for all the replies! Didn’t expect so many tips.

Tried aiohttp + proxies and it’s way faster. Still hitting some blocks tho—might give Scrapy a shot next.

Anyone got a favorite proxy provider that’s budget-friendly?

Also, Playwright looks dope for JS sites. Gonna test it tonight.

Cheers!