Proxy Community
What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl - Printable Version

+- Proxy Community (https://proxycommunity.com/forum)
+-- Forum: Use Case (https://proxycommunity.com/forum/forum-use-case)
+--- Forum: Web Scraping (https://proxycommunity.com/forum/forum-web-scraping)
+--- Thread: What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl (/thread-what-s-the-best-way-to-build-a-web-crawler-in-python-from-scratch-or-how-can-i-optimize-my-web-crawl)

Pages: 1 2 3


“” - cloakNomad_77 - 03-04-2025

For storing data, skip CSV if you’re doing big stuff. PostgreSQL or MongoDB scale better.

Use `pymongo` for MongoDB—super easy to dump JSON data.

```python
from pymongo import MongoClient
client = MongoClient('localhost', 27017)
db = client['scraped_data']
db.pages.insert_one({'url': '...', 'content': '...'})
```


“” - FirewallVoyagerX - 04-04-2025

Wow, thanks for all the replies! Didn’t expect so many tips.

Tried aiohttp + proxies and it’s way faster. Still hitting some blocks tho—might give Scrapy a shot next.

Anyone got a favorite proxy provider that’s budget-friendly?

Also, Playwright looks dope for JS sites. Gonna test it tonight.

Cheers!