How do you create a web crawler from scratch? Need tips and guidance! or Looking to build a web crawl

Wow, didn’t expect so many replies!

Tried the `requests` + `BeautifulSoup` combo and it’s way less scary than Scrapy. Still getting blocked on some sites though—guess I’ll test those proxy tips next.

Anyone got a favorite free proxy service, or is it worth paying for one?

Also, Playwright looks dope. Gonna check that out too. Thanks y’all!