What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl

22 Replies, 1187 Views

For storing data, skip CSV if you’re doing big stuff. PostgreSQL or MongoDB scale better.

Use `pymongo` for MongoDB—super easy to dump JSON data.

```python
from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db = client['scraped_data']
db.pages.insert_one({'url': '...', 'content': '...'})
```

Messages In This Thread



Users browsing this thread: 1 Guest(s)