![]() |
|
What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl - Printable Version +- Proxy Community (https://proxycommunity.com/forum) +-- Forum: Use Case (https://proxycommunity.com/forum/forum-use-case) +--- Forum: Web Scraping (https://proxycommunity.com/forum/forum-web-scraping) +--- Thread: What's the best way to build a web crawler in Python from scratch? or How can I optimize my web crawl (/thread-what-s-the-best-way-to-build-a-web-crawler-in-python-from-scratch-or-how-can-i-optimize-my-web-crawl) |
“” - cloakNomad_77 - 03-04-2025 For storing data, skip CSV if you’re doing big stuff. PostgreSQL or MongoDB scale better. Use `pymongo` for MongoDB—super easy to dump JSON data. ```python from pymongo import MongoClient client = MongoClient('localhost', 27017) db = client['scraped_data'] db.pages.insert_one({'url': '...', 'content': '...'}) ``` “” - FirewallVoyagerX - 04-04-2025 Wow, thanks for all the replies! Didn’t expect so many tips. Tried aiohttp + proxies and it’s way faster. Still hitting some blocks tho—might give Scrapy a shot next. Anyone got a favorite proxy provider that’s budget-friendly? Also, Playwright looks dope for JS sites. Gonna test it tonight. Cheers! |