I’ve used this crawler for scraping around 50k pages, and it held up surprisingly well.
Speed-wise, it’s decent—not the fastest, but definitely not slow either. For 10k pages, it took a few hours, but that’s with proxies and delays to avoid bans.
Speaking of bans, yeah, it handles rate limits pretty well. Just make sure you tweak the settings and use rotating proxies. I’d recommend Bright Data or Oxylabs for proxies if you’re doing heavy scraping.
Speed-wise, it’s decent—not the fastest, but definitely not slow either. For 10k pages, it took a few hours, but that’s with proxies and delays to avoid bans.
Speaking of bans, yeah, it handles rate limits pretty well. Just make sure you tweak the settings and use rotating proxies. I’d recommend Bright Data or Oxylabs for proxies if you’re doing heavy scraping.