What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl

For storing data, skip CSV if you’re doing big stuff. PostgreSQL or MongoDB scale better.

Use `pymongo` for MongoDB—super easy to dump JSON data.

```python
from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db = client['scraped_data']
db.pages.insert_one({'url': '...', 'content': '...'})
```

Wow, thanks for all the replies! Didn’t expect so many tips.

Tried aiohttp + proxies and it’s way faster. Still hitting some blocks tho—might give Scrapy a shot next.

Anyone got a favorite proxy provider that’s budget-friendly?

Also, Playwright looks dope for JS sites. Gonna test it tonight.

Cheers!