I’ve tried cloutgist scrapy, and honestly, it’s a mixed bag. It’s got some cool features, but the setup is a bit of a headache compared to Scrapy or BeautifulSoup.
If you’re scraping small to medium-sized projects, it works fine, but for large-scale stuff, it tends to slow down after a few thousand requests.
I’d recommend checking out Playwright or Puppeteer if you’re looking for something more robust. They handle dynamic content way better, and the setup is way smoother.
If you’re scraping small to medium-sized projects, it works fine, but for large-scale stuff, it tends to slow down after a few thousand requests.
I’d recommend checking out Playwright or Puppeteer if you’re looking for something more robust. They handle dynamic content way better, and the setup is way smoother.