How do you create a web crawler from scratch? Need tips and guidance! or Looking to build a web crawl

how do you create a web crawler that scales?

Don’t.

Seriously, unless you NEED to, use an existing tool.

But if you insist, look into `aiohttp` for async crawling. And for the love of god, respect `robots.txt`.