How Effective Is This Crawler at Scraping Dynamic Websites? or Can This Crawler Handle Large-Scale Da

22 Replies, 903 Views

This crawler’s pretty reliable for large jobs, but it’s not a "set and forget" tool. You’ll need to babysit it a bit.

Speed is decent, but if you’re in a hurry, maybe look at Scrapy + Scrapinghub for more control.

For proxies, I’ve had good luck with GeoSurf.

Messages In This Thread



Users browsing this thread: 1 Guest(s)